Hacker News new | ask | show | jobs
by JesseAldridge 3632 days ago
I think Sublime Text should be able to handle fairly large json files without a problem.

If not, I think I would just split data into multiple smaller files. Something like the python code below should work, assuming the file can fit in memory. If not, I assume you can find some json lib that can work in streaming mode and then do the same thing.

    import json, os

    json_input = '''{
    "foo": 1,
    "bar": 2,
    "baz": 3,
    "bug": 4,
    "thing": 5
    }'''

    entries_per_group = 2

    if not os.path.exists('sub_files'):
      os.mkdir('sub_files')

    main_d = json.loads(json_input)
    iter = main_d.iteritems()
    for group_count in range(10 ** 6):
      sub_d = {}
      try:
        for _ in range(entries_per_group):
          k, v = iter.next()
          sub_d[k] = v
      except StopIteration:
        break
      finally:
        json_output = json.dumps(sub_d, indent=2)
        with open(os.path.join('sub_files', '{}.json'.format(group_count)), 'w') as f:
          f.write(json_output)