I’ve been adding more “stuff” from my life to this website. I use services like Instagram, Flickr, Mastodon, Foursquare, Facebook, StackOverflow, GitHub which have public-facing components. In the spirit of the #indieweb PESOS I got the notion to import my over-a-decade worth of Foursquare and Swarm checkins to my site. I don’t have a particular goal in mind, though it might be nice to add location context to some of my photos. I’ve been hesitant to do this because really, sharing realtime data give me an ooky feeling.
But it is my data, so if anyone’s going to do anything with it it ought to be me.
The first thing I looked at was whether I could use Foursquare APIs to get this data. I set up a developer account and played a bit, but once I hit the API key to OAuth exchange step it started to occur to me that these APIs are not designed for an individual to access their own data. It’s really about entities who want to explore the whole corpus of location data and activity across the planet. My needs are far more modest.
It became clear what I wanted was an export of my own data. So, I requested it, which I could do from the Swarm App:
A few days later, I got the data. It’s a zip file with a pile of files in it, here’s a listing:
checkins7.json checkins1.csv checkins1.json checkins2.csv checkins2.json checkins3.csv checkins3.json checkins4.csv checkins4.json checkins5.csv checkins5.json checkins6.csv checkins6.json checkins7.csv comments.csv comments.json expertise.csv expertise.json paymentProfiles.csv paymentProfiles.json photos.csv photos.json readme.html tastes.csv tastes.json tips.csv tips.json unconfirmed_visits.csv unconfirmed_visits.json uniqueDevices.csv uniqueDevices.json users.csv users.json venueRatings.csv venueRatings.json visits.csv visits.json pix/
And a few hours after that, after writing some PHP code, I had some checkins displaying to my website. I wrote code to parse the JSON files and handle the edge cases. Checkins come 1000 to a file, and some checkins are arbitrary text, and don’t have a true “venue” – they have a type of “venueless” – and if you checkin multiple times per day you get a checkin without venue data. All the little decisions made by Foursquare programmers years ago that impact the shape of the data here and now. Data has lots of wrinkles, and resolving what to do with them is the role of the programmer.
So I’ve added a display of the checkins to single blog posts if they exist, and single days, and also to month views. I’m considering whether I want to more closely correlate the data with my own posts, or include photos I posted over the years. I may yet do that. It might be fun to correlate beach and pool images on my blog with the location data. Actually come to think of it in some cases there is Instagram location data I could capture for those photos.
Aside: Instagram is hostile to programmatically asking “hey, what location was it for that photo?” I would probably use instaLooter to do that kind of work. I like that tool a lot. That project and its programmers keeps up with the little war Instagram wages on people trying to backup their photos and work with Instagram data. Back in 2020 I did that: Instalooter to WordPress
It was fun to explore the thousands of checkins and see lots of my life unfold. I remember seeing that movie! That was a fun baseball games I’ve not imported the data as WordPress data at this point. I exported it all to html files which I can display as needed. So I have a folder full of html files named by date. I am considering whether I want to import the data in that way. I’ve shied away from relying on adding lots of metadata to my WordPress posts but maybe if I’m expecting the site to have richer content I need to embrace that kind of change.
So here’s an example, from May of this year:
It’s my data. It’s my website. I believe we as users of these services have an absolute right to repurpose our own data as we see fit. As a programmer, I am proud to have capability to do so.
Here’s the code I wrote today: foursquare-export-to-html
