[kfullert]

Not fully (ie in production) however a new side-project I'm working on which has many processes I'm building each as a micro-service, with RabbitMQ handling the messaging, Bunny to handle message processing and Sinatra/Faraday for where the messages are too big to send via MQ (ie when I process a 2.3Gb JSON file, I send the filename to a queue, which triggers a Faraday request by the worker to a Sinatra app to download/process the actual file)

About the only thing I need to really work on is directory/registration (at the moment I'm using .env files with the URLs to different micro-services in each micro-service, mainly as it's PoC and I'm running it all on a single box)

[_RPM]

I'm just curious. When dealing with JSON files that are 2.3Gb, what programming language do you use to parse it?

[kfullert]

Ruby at the moment - my original (monolithic Rails) v0 version takes about 5 hours to process it and create database records with ActiveRecord - that's about 600,000 records - the microservices are using Oj for the JSON parsing and raw pg queries for database (so not a real comparison at the moment)

[earless1]

What is your app doing that requires a 2.3Gb JSON file?

[kfullert]

It's working with the Schedule data from National Rail (under their Open Data - http://nrodwiki.rockshore.net/index.php/SCHEDULE) - the "full" file for initially populating your database is around 2.3Gb uncompressed JSON (nightly updates range from 1Mb to 31M looking at the files I've got here)