Hacker News new | ask | show | jobs
by tedivm 1667 days ago
Athena is not BigQuery. Athena is just a fancy wrapper around Hive (you can see this right from the log output) and just runs map reduce over your S3 data. It's a great tool for what it is.

BigQuery is a full database. It is significantly faster than running anything from Athena. The closest comparison on AWS is Redshift.
2 comments
ideonode 1667 days ago
Athena is more of a wrapper over Presto / Trino than Hive. It does not run map reduce per se over your data - its a SQL engine.
link
pdelgallego 1667 days ago
AWS Glue Data catalog is Hive, and as you mentioned, Athena is a wrapper of Presto
link
mdasen 1666 days ago
I think Athena has a lot of value (especially given its pricing), but you're not wrong that it has limitations.

Getting data into Athena isn't something that is just done for you. Athena just takes what you've put on S3 and queries over it - and leaves getting it onto S3 (and into an efficient format) as an exercise for the reader.

Athena's speed varies a lot depending on what format you put things in. Querying over CSVs will mean that you're slow and reading a lot of data. Querying over ORC (column-store) files is pretty quick.

The big thing is Athena's pricing. They price it on how much data you read in reality not how much data would be read if things weren't optimized. BigQuery charges you based on how much data would be read if it weren't optimized. With BigQuery, an integer is always 8 bytes. It doesn't matter if they're able to optimize it down to nothing using RLE (run length encoding). You still pay the full 8 bytes. If your ORC files make that integer column tiny, you get the benefit of that.

BigQuery is great, but Athena's pricing is a lot cheaper given that you get to benefit from any storage optimization you do.

Out of curiosity, how have you used Athena that you're seeing it be so much slower? In my experience, BigQuery is faster (maybe 2x faster), but I've been using column-oriented data with Athena. If you're using CSVs with Athena, it will be way slower than BigQuery.

I'm always a little surprised that AWS doesn't build Athena out more, but I guess if they did they'd want money and margin for the value add. Still, Athena is a pretty decent serverless Presto and Presto can work pretty well over data in column formats.

link