| The more concrete feedback is in the previous post (you commented on that one too). But I will elaborate here: On 8/4/15, I received a message from an Asana representative via Stack Overflow Careers. After a few back-and-forths discussing specifics about whether I was a good fit, I then agree to do the take-home test. I received the take home test on 8/7/15. I emailed back on 8/8/15 with a verifiable error in a timestamp column of an AWS-hosted Postgres database that was configured with some test data (data resembling some certain kinds of user data for modeling what an "engaged" user is, and what factors predict when a user will be "engaged"). I included code which could be run to demonstrate the erroneous data. On 8/9/15 I wrote in again stating that I had created a workaround for the erroneous timestamp data, since the only way I could complete the exam was to do it on those particular days (I couldn't wait around for the exam data to be fixed). I submitted my solution on 8/10/15. I got an email on 8/11/15 with a note from someone in engineering apologizing for the timestamp data error, but also saying it shouldn't affect my solution. The recruiter who sent that email also said that my submission was currently being evaluated. Then I waited a week before hearing anything. On 8/18/15 I got a response apologizing for the delay and saying that I was invited for the next round interview. The email said I would be speaking with a person named Jack (maybe you?) a few days later (8/21/15 at 4:00 pm EST). But that call actually included two data scientists over one conference call line. Both interviewers effectively made no comment on my software choices. In fact, when I spoke about how I cared about software design and best practices, even when working on a rapid prototype, it actually sounded as if the interviewers thought this was a bad thing and there was some awkward silence. I brought up examples of how I had worked in a very fast paced environment before (quant finance) and my team had learned some hard lessons that the naive approach of letting data scientists just write scripts for ad hoc modeling, assuming that engineering will "productionize" it later, tends to lead to some bad failure cases, and that it's far more efficient over time to simply require data scientists to construct rapid prototypes, even from the first line of exploratory, ad-hoc code, that adhere to many best practices and are only a short distance from "production" even in their first version. Then they asked me some vague and hard-to-decipher-over-a-conference-call questions about how to measure the accuracy for the random forest model I had fitted, and about my conclusions regarding which predictors were most efficacious. Despite having used random forests for published research results and even having taught part of a course on random forests when I was a grad student, I could not make heads nor tails of what exact accuracy metric the interviewer was fishing for. I am sure that if such a question was asked in-person, with some pen and paper, or something where it's not just the vague descriptions of some stranger I've never met before, then we could have sorted it out. But the communication was just too poor over the conference call for this kind of question. Given that I had to solve the problem in my spare time while conducting all the other stuff in my life, I felt it was quite reasonable for me to take a very conservative approach: fit a very simple model, don't waste time chasing down parameter optimizations (like depth of the trees or some huge set of features). Just walk through some very straightforward analysis, use some straightforward features, produce some common diagnostic curves (which showed a huge bias in the data labels -- almost all example users were not "engaged" ), and draw only conservative conclusions (e.g. with such severe bias in the toy data set, it's pretty hard to conclusively say anything, so don't pretend you can say more than you really can, don't recommend strong conclusions, and leave it at that). The point is to show basically a mock-up of the end-to-end workflow, from querying the data, to constructing the features, to fitting the model, to assessing the fit. The point is clearly not to do that in some kind of insanely accurate way when you're talking about a busy adult doing it in their spare time over a few days. The two interviewers had some gripes with the specifics of my accuracy measures (which seemed really misplaced, again for an analysis done in a busy adult's spare time using erroneous toy data). |
I was actually fairly detailed in my description of some math ideas for these things, but the whole conversation sounded like someone sucked the air out of it. The interviewers kept trying to get me to talk more specifically about aspects of the newsfeed that actually occur in Facebook's newsfeed. When I told them I don't have a Facebook account and don't know what the modern version of the newsfeed does (which is true, I don't), they sounded like they just lost all interest in talking to me. My hunch was that they wanted me to possibly talk about A/B testing, but this is kind of silly. They definitely didn't ask me about A/B testing. It was like they wanted me to guess that phrase. Plus, at least for frequentist A/B testing, there are good reasons to believe a lot of it is junk science and a lot of applied mathematicians would not consider this a reasonable idea at all. I guess I could have talked a little about Bayesian A/B testing, but my mind associated their problem with network modeling and seasonality effect modeling, not A/B testing. (I can't be sure they were focused on A/B testing -- it's just what I independently thought after the fact when struggling to understand the outcome since no one gave any feedback).
Overall, I could tell it went badly, but I could not understand what the two interviewers were trying to get me to say. As someone who has worked on a lot of math modeling problems, from undergrad and on through grad school, I thought my way of breaking down and describing ideas for the newsfeed example was perfectly on-point and technically useful. To boot, the job I was interviewing for did not (at least as it was explained to me) have anything to do with ranking problems, A/B testing, or newsfeed-like work at all.
That was on 8/21/15. I then heard no response of any kind, until 8/28/15 when I got a form letter rejection email that simply said at that point, Asana did not want to continue the process, and gave no further details.
From start to finish it went from 8/4 to 8/28, so almost a month. I scrambled to put in a lot of hours to solve the take-home test in my spare time over a short window, and then went through two different full week long periods of silence (one after my code test was being reviewed, and one after the vague/weird two-on-one phone interview), only to receive no feed back at all on my software solution, limited and tangential feedback on my actual data science approach (feedback which didn't seem at all appropriate for something a busy adult would put together over a couple of days), and no feedback at all about the decision to reject me based on the interviews.
The overall experience left me feeling like Asana asks very vague questions and reserves the right to be dissatisfied if you don't identify the magic answer to the vague question. It also made me feel that Asana does not act in a timely manner to review code submissions or to provide interview feedback, even though Asana did require me (despite being busy outside of interviews) to complete a code test in a timely manner. It also sounded like Asana might have some data science employees who act a bit like "gatekeepers" in the interview process -- they felt a need to assert that they are better than the candidate, even if it means bringing some discussion up about specific accuracy criteria or something when it may not really be very appropriate for a quick solution to a poorly specified problem.
It's a bit like someone talking about Haskell programming for an interview, successfully solving some Monad problems in a quick way given the time constraints, and then getting grilled about unification algorithms in the Hindley-Milner type system. It's just to let the existing employee who is conducting the interview feel superior and act like it is some great filter on the incoming talent pool, when really it's mostly unrelated minutiae.
As I mentioned in the other comment, I later learned the extent to which Asana supports Agile-focused systems for enterprise project management, and this actually saddened me a lot. I always liked Asana because it embodied a type of minimalist efficiency that is fundamentally incompatible with the bureaucratic wastefulness of Agile/Scrum/Kanban/"lean" buzzword systems. I had high hopes that Asana would not be just another one of those cargo cult management junk systems, and maybe it still will avoid that fate, but agreeing to support Agile doesn't bode well.
So ultimately it was a good thing I was rejected, though I do not feel it was a fair reflection of my engineering effort on the exam, and I certainly don't feel that the evaluation was done in a way that showed respect for the amount of time and effort that I took to complete the task.