[apohn]

This may seem like really basic and obvious advice, but I’ve always found it helpful to think about solving a much smaller problem, solving that, and then going to management with that as a guidepost for building out the larger project. Putting “Build Data+ML Pipeline from scratch” + “In 2 weeks” is a great way to lose all hope and feel like you are drowning.

For example, PaulHoule mentioned Enron emails. Could you build a pipeline to ingest those emails and provide sentiment or some basic text analysis? That would give you a deliverable (so management/the team sees you are working towards something) you can quantify into sprints. Make sure you are building something that could be development into a production ready pipeline, and not just a toy script you can put out in a day or two.

After that is built, I’d use that as a guidepost for what you need to build. Show them the project and then use that to define the different stages you need to build for the production application. Include risks, blocking issues, and what you need from other team members. For example, you can point out that after a certain point in developing the pipeline you simply cannot move forward without data or emails.

Management/The team is going to come back to you with one of the following. 1) That is a good plan and we like/dislike the timeline. 2). That is a bad plan, change it. 3). Leave us alone, we hired you so we don’t have to deal with this stuff.

If they do 3) that’s bad. If they do 2), you should evaluate who is being unrealistic – you or them. At my last job we lost great data scientists because lots of other people thought Stats/ML was magic that could be done with a simple R/Python script. The end result was severely rushed work done to unrealistic standards, resulting in massive technical debt – some projects were just a house of cards.