Hacker News new | ask | show | jobs
by siganakis 4358 days ago
From my reading of the getting started guide, it looks like it treats your "working" directory as a git repository, with each run basically doing a commit & push.

This means that your data (in files) needs to be in the working directory, and is versioned along side your code. Sounds pretty cool, but I am not sure how it would scale for large / constantly changing data sets.

1 comments

siganakis's explanation is correct, with a minor caveat [0]. We do some clever things (diffing, compressing), but if large amounts of data are always changing, the network transfer will take time. We have folks comfortably using Domino with about 50GB of data in their project directories. And it's only the amount of data in the current revision -- not cumulative across all revisions -- that matters. Anyway, if you have a use case with more than ~50GB, let us know, we're eager to engineer a more advanced solution -- just haven't had the need yet ;)

If your code pulls data from a database (e.g., kelv's test cases), one option is to save the DB snapshot out to a file when the code runs. After we finish executing your code, we snapshot all the new/changed files in your working directory (we call those the "results" of the run). Using this approach, you'd have a record of the DB snapshot for each run of your tests. But, again... if your snapshot is more than 10s of GB, the network transfer time could get annoying.

[0] We treat the working directory as a git repo, but since git breaks down with large files, we only use it to track your directory structure; we store actual file contents elsewhere.