[jacklionheart]

Hi there! That sounds bad. Could you let me know what's giving you the impression that that's Asana's goal?

We really try to explicitly avoid that kind of interview evaluation, and put strong emphasis on Communication, User empathy, and Learning, per the guide. But definitely "did they say the right answer?" is an easy trap to fall into so its entirely possible that some interviewers are doing it -- would love any concrete feedback you could give to help us avoid it in the future.

[p4wnc6]

The more concrete feedback is in the previous post (you commented on that one too). But I will elaborate here:

On 8/4/15, I received a message from an Asana representative via Stack Overflow Careers. After a few back-and-forths discussing specifics about whether I was a good fit, I then agree to do the take-home test.

I received the take home test on 8/7/15. I emailed back on 8/8/15 with a verifiable error in a timestamp column of an AWS-hosted Postgres database that was configured with some test data (data resembling some certain kinds of user data for modeling what an "engaged" user is, and what factors predict when a user will be "engaged"). I included code which could be run to demonstrate the erroneous data.

On 8/9/15 I wrote in again stating that I had created a workaround for the erroneous timestamp data, since the only way I could complete the exam was to do it on those particular days (I couldn't wait around for the exam data to be fixed).

I submitted my solution on 8/10/15. I got an email on 8/11/15 with a note from someone in engineering apologizing for the timestamp data error, but also saying it shouldn't affect my solution. The recruiter who sent that email also said that my submission was currently being evaluated.

Then I waited a week before hearing anything. On 8/18/15 I got a response apologizing for the delay and saying that I was invited for the next round interview.

The email said I would be speaking with a person named Jack (maybe you?) a few days later (8/21/15 at 4:00 pm EST). But that call actually included two data scientists over one conference call line.

Both interviewers effectively made no comment on my software choices. In fact, when I spoke about how I cared about software design and best practices, even when working on a rapid prototype, it actually sounded as if the interviewers thought this was a bad thing and there was some awkward silence. I brought up examples of how I had worked in a very fast paced environment before (quant finance) and my team had learned some hard lessons that the naive approach of letting data scientists just write scripts for ad hoc modeling, assuming that engineering will "productionize" it later, tends to lead to some bad failure cases, and that it's far more efficient over time to simply require data scientists to construct rapid prototypes, even from the first line of exploratory, ad-hoc code, that adhere to many best practices and are only a short distance from "production" even in their first version.

Then they asked me some vague and hard-to-decipher-over-a-conference-call questions about how to measure the accuracy for the random forest model I had fitted, and about my conclusions regarding which predictors were most efficacious. Despite having used random forests for published research results and even having taught part of a course on random forests when I was a grad student, I could not make heads nor tails of what exact accuracy metric the interviewer was fishing for. I am sure that if such a question was asked in-person, with some pen and paper, or something where it's not just the vague descriptions of some stranger I've never met before, then we could have sorted it out. But the communication was just too poor over the conference call for this kind of question.

Given that I had to solve the problem in my spare time while conducting all the other stuff in my life, I felt it was quite reasonable for me to take a very conservative approach: fit a very simple model, don't waste time chasing down parameter optimizations (like depth of the trees or some huge set of features). Just walk through some very straightforward analysis, use some straightforward features, produce some common diagnostic curves (which showed a huge bias in the data labels -- almost all example users were not "engaged" ), and draw only conservative conclusions (e.g. with such severe bias in the toy data set, it's pretty hard to conclusively say anything, so don't pretend you can say more than you really can, don't recommend strong conclusions, and leave it at that). The point is to show basically a mock-up of the end-to-end workflow, from querying the data, to constructing the features, to fitting the model, to assessing the fit. The point is clearly not to do that in some kind of insanely accurate way when you're talking about a busy adult doing it in their spare time over a few days.

The two interviewers had some gripes with the specifics of my accuracy measures (which seemed really misplaced, again for an analysis done in a busy adult's spare time using erroneous toy data).

[p4wnc6]

After that, the interviewers asked the questions about how to develop a news feed ranking algorithm at a super vague, high level. I spoke about how you would need to account for seasonality or cyclic effects -- e.g. election news would clearly be ranked higher in newsworthiness when approaching important elections, or the same idea for sports-related articles, etc. I talked about trying to determine which news articles are the most clicked-on by peers in your network, possibly using some centrality measures to rate certain users more as "trend setters" and use their news viewing habits as guides for what to show to others.

I was actually fairly detailed in my description of some math ideas for these things, but the whole conversation sounded like someone sucked the air out of it. The interviewers kept trying to get me to talk more specifically about aspects of the newsfeed that actually occur in Facebook's newsfeed. When I told them I don't have a Facebook account and don't know what the modern version of the newsfeed does (which is true, I don't), they sounded like they just lost all interest in talking to me. My hunch was that they wanted me to possibly talk about A/B testing, but this is kind of silly. They definitely didn't ask me about A/B testing. It was like they wanted me to guess that phrase. Plus, at least for frequentist A/B testing, there are good reasons to believe a lot of it is junk science and a lot of applied mathematicians would not consider this a reasonable idea at all. I guess I could have talked a little about Bayesian A/B testing, but my mind associated their problem with network modeling and seasonality effect modeling, not A/B testing. (I can't be sure they were focused on A/B testing -- it's just what I independently thought after the fact when struggling to understand the outcome since no one gave any feedback).

Overall, I could tell it went badly, but I could not understand what the two interviewers were trying to get me to say. As someone who has worked on a lot of math modeling problems, from undergrad and on through grad school, I thought my way of breaking down and describing ideas for the newsfeed example was perfectly on-point and technically useful. To boot, the job I was interviewing for did not (at least as it was explained to me) have anything to do with ranking problems, A/B testing, or newsfeed-like work at all.

That was on 8/21/15. I then heard no response of any kind, until 8/28/15 when I got a form letter rejection email that simply said at that point, Asana did not want to continue the process, and gave no further details.

From start to finish it went from 8/4 to 8/28, so almost a month. I scrambled to put in a lot of hours to solve the take-home test in my spare time over a short window, and then went through two different full week long periods of silence (one after my code test was being reviewed, and one after the vague/weird two-on-one phone interview), only to receive no feed back at all on my software solution, limited and tangential feedback on my actual data science approach (feedback which didn't seem at all appropriate for something a busy adult would put together over a couple of days), and no feedback at all about the decision to reject me based on the interviews.

The overall experience left me feeling like Asana asks very vague questions and reserves the right to be dissatisfied if you don't identify the magic answer to the vague question. It also made me feel that Asana does not act in a timely manner to review code submissions or to provide interview feedback, even though Asana did require me (despite being busy outside of interviews) to complete a code test in a timely manner. It also sounded like Asana might have some data science employees who act a bit like "gatekeepers" in the interview process -- they felt a need to assert that they are better than the candidate, even if it means bringing some discussion up about specific accuracy criteria or something when it may not really be very appropriate for a quick solution to a poorly specified problem.

It's a bit like someone talking about Haskell programming for an interview, successfully solving some Monad problems in a quick way given the time constraints, and then getting grilled about unification algorithms in the Hindley-Milner type system. It's just to let the existing employee who is conducting the interview feel superior and act like it is some great filter on the incoming talent pool, when really it's mostly unrelated minutiae.

As I mentioned in the other comment, I later learned the extent to which Asana supports Agile-focused systems for enterprise project management, and this actually saddened me a lot. I always liked Asana because it embodied a type of minimalist efficiency that is fundamentally incompatible with the bureaucratic wastefulness of Agile/Scrum/Kanban/"lean" buzzword systems. I had high hopes that Asana would not be just another one of those cargo cult management junk systems, and maybe it still will avoid that fate, but agreeing to support Agile doesn't bode well.

So ultimately it was a good thing I was rejected, though I do not feel it was a fair reflection of my engineering effort on the exam, and I certainly don't feel that the evaluation was done in a way that showed respect for the amount of time and effort that I took to complete the task.

[jacklionheart]

Yep, I'm that Jack. I remember you now. Thanks for the clear picture of what it's like to be on the candidate side. Efficiency is clearly a goal, and we clearly failed here given that it took us a week to get back to you after the phone interivew. I do think your interview was an outlier in that regard, but that doesn't excuse it.

Re: the questions themselves, although I don't agree with your characterization of what's going on or what we're looking for, I definitely think there's a lot of room to improve on our data-science screening. I'll send you an email with more re: what specifically came up in your interview that made us reject.

Again, thanks for taking the time to share this really explicit feedback.

(For the record, I don't really know what you're referring to re: Asana and Agile.)

[p4wnc6]

Hopefully this thread's not too stale yet to escape notice.

I just wanted to add that the user above, Jack, absolutely made good on what he said.

He sent me some very well-written, extremely professional, and constructive feedback about my interview process. I am able to see a lot of things I can improve and it will help me for future interviews. I think it also highlights some things about the process that Asana can improve, and by all evidence of Jack's sincerity in the feedback, I believe Asana is very interested in ways of improving.

Don't let anyone say that all I do on Hacker News is grumble about open-plan offices... in this case I am expressing my sincere gratitude and appreciation for Jack's feedback.

If I had received this kind of response in a timely manner when Asana decided not to continue the interview process with me, I would have walked away with a significant positive feeling about interviewing with them.