How much did i screw up?

[LELISOSKA]

i know there are alot of programmers on this board, so tell me me how on a scale of 1 to 10 how much of a moron i am for using node.js and mongodb to do the following: scrape 8 different social outlet / portfolio apis sync the the scraped data with 3 different post model schemas as well as a user model using a key store database mongodb all of that wrapped in promise library to avoid callbacks. scrape data live and based on certian queries, save, sync etc. everything wrapped in promises and i dont even know what will happen if two users run the same query async. why would i ever use node.js instead of something like python or ruby or whatever and just split the stuff into threads instead of having to deal with retarded javascript async??

the only reason i did it is because javascript was the first thing i learned besides some c++ or whatever in a few semesters of college.

should i be using a normal synchronous language like c# or java or something?

[ralusek]

None of the problems you have seem to be related to language, let alone whether it's asynchronous, synchronous, multithreaded or single threaded. What you've described here sounds simple enough from an outside perspective, so it's more than likely just time to reconsider your architecture.

So let's talk about solving this with NodeJS. First of all, you've allowed this to get overwhelming to you because you consider this whole thing to be a monolithic service, when it's really just time to break it into modules. It sounds like you have a scraper, a parser, some application which needs the scraped data, and you have a web API for that application. Cool, easy enough.

So the first thing I would do is pull out your scraper into a completely different service which has nothing to do with your API. Just have it be a javascript class which takes some configurations to manage the scraping settings, and that's about it. All you really need it to do is have a method which takes some input which tells it what site to scrape, and it will then give you the output.

Cool, now you need a parser. You need to go through the html returned by your scraper and format it as JSON. Fortunately, Javascript has a lot of tools very well suited for dealing with HTML. A tool I like is called Cheerio, which makes it very easy to parse HTML and get the data you need from it. So you have a parser now.

All that's left is integrating with your application now. Based off of you saying that you need to "scrape data live and based on certain queries", what we want is to expose some routes in your application which make that possible.

If you tell me a little bit more about what your application does, I can give you some idea of how I would structure this from here.

[coreyp_1]

You never "screw up" by learning. For that matter, you learned a lot about javascript and asynchronous programming. Furthermore, you would probably now (more than the average programmer) appreciate the advances of ES6 and the promises/generator pattern.

As for what you "should" be using, there is no right or wrong answer. The biggest metric is whether or not the language is a good fit for the task. The only thing that JS is bad at, IMO, is CPU-intensive tasks. Aside from that, it may be perfectly adequate for the job.

The only other factors would be if there were a language that offered features (or libraries) that you need.

[niftich]

Does your product work? If yes, you're not doing too bad.

Under what circumstances does your product break? Test this, don't just speculate!

Are you finding that you're having to jump through too many hoops to get the code to do what you want, and are wishful for a different language/platform/API? If you're fighting against your current tools, investigate refactoring, or replacing small components to see if they help.

Never throw anything that worked away! Version control, version control!

[bigiain]

Depending on what assurances and promises you've made to other people about the robustness of your current solution, I reckon there's a high likelyhood you haven't "screwed up" at all. You've investigated your problem space, found some workable solutions, and got some proof-of-concept code running.

Writing something in a language you know is always better than not writing anything in a "better" language that you don't know. For _you_, Javascript was the right language, at least for your initial explorations. You're now a long way ahead of where you started, with new-found knowledge about some of the edge case problems and implementation details that you never had when you started.

Javascript async is difficult (I'd hesitate to jump all the way to "retarded"), mostly because async itself is difficult (though, to be honest, a lot of people consider JS to have implemented it badly as well, but that's kind of secondary).

Having said that, there are going to be similar downsides to doing it in Python, Ruby, Haskel, Erlang, Go, Groovy, Befunge, Brainfuck, Assembler, Perl - that's why there isn't just one language, they all have different compromises, different learning curves, different support communities, different library availability and robustness, different expertise availability...

Try not to ask "how much have I screwed up?", but instead ask "what has this version taught me?", and "which of the decisions I made would I have made differently if I knew at the beginning what I knew now?"

Bottom line - your project _can_ be written in Javascrip. It's almost certainly significantly less complex than Gmail or Googlemaps. It just might take some very senior level Javascript developers to get there.

Don't think that jumping from college-project-level Javascript to I've-read-a-bit-about-it-level Ruby or Python is going to magically make the hard things easy though.

[LELISOSKA]

its a personal project and i'm the only one working on it so i have to figure it out on my own but thanks for the advice

[bwackwat]

If you have a working implementation and have yielded moderate results, then you have earned "success".

In this case I get the feeling you want to use a different set of technologies to either improve performance or ease extensibility.

How much did you screw up? 7/10. You have found that node.js and mongodb are not the most mature technologies and would likely find python + sql (mssql, sqlite, postgresql, whatever) ... nicer.

[asimuvPR]

You only know how to build a good program after you've built it. Go back and re do it. Repeat as much as possible.