TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

How much did i screw up?

4 pointsby LELISOSKAalmost 9 years ago
i know there are alot of programmers on this board, so tell me me how on a scale of 1 to 10 how much of a moron i am for using node.js and mongodb to do the following: scrape 8 different social outlet &#x2F; portfolio apis sync the the scraped data with 3 different post model schemas as well as a user model using a key store database mongodb all of that wrapped in promise library to avoid callbacks. scrape data live and based on certian queries, save, sync etc. everything wrapped in promises and i dont even know what will happen if two users run the same query async. why would i ever use node.js instead of something like python or ruby or whatever and just split the stuff into threads instead of having to deal with retarded javascript async??<p>the only reason i did it is because javascript was the first thing i learned besides some c++ or whatever in a few semesters of college.<p>should i be using a normal synchronous language like c# or java or something?

6 comments

ralusekalmost 9 years ago
None of the problems you have seem to be related to language, let alone whether it&#x27;s asynchronous, synchronous, multithreaded or single threaded. What you&#x27;ve described here sounds simple enough from an outside perspective, so it&#x27;s more than likely just time to reconsider your architecture.<p>So let&#x27;s talk about solving this with NodeJS. First of all, you&#x27;ve allowed this to get overwhelming to you because you consider this whole thing to be a monolithic service, when it&#x27;s really just time to break it into modules. It sounds like you have a scraper, a parser, some application which needs the scraped data, and you have a web API for that application. Cool, easy enough.<p>So the first thing I would do is pull out your scraper into a completely different service which has nothing to do with your API. Just have it be a javascript class which takes some configurations to manage the scraping settings, and that&#x27;s about it. All you really need it to do is have a method which takes some input which tells it what site to scrape, and it will then give you the output.<p>Cool, now you need a parser. You need to go through the html returned by your scraper and format it as JSON. Fortunately, Javascript has a lot of tools very well suited for dealing with HTML. A tool I like is called Cheerio, which makes it very easy to parse HTML and get the data you need from it. So you have a parser now.<p>All that&#x27;s left is integrating with your application now. Based off of you saying that you need to &quot;scrape data live and based on certain queries&quot;, what we want is to expose some routes in your application which make that possible.<p>If you tell me a little bit more about what your application does, I can give you some idea of how I would structure this from here.
coreyp_1almost 9 years ago
You never &quot;screw up&quot; by learning. For that matter, you learned <i>a lot</i> about javascript and asynchronous programming. Furthermore, you would probably now (more than the average programmer) appreciate the advances of ES6 and the promises&#x2F;generator pattern.<p>As for what you &quot;should&quot; be using, there is no right or wrong answer. The biggest metric is whether or not the language is a good fit for the task. The only thing that JS is bad at, IMO, is CPU-intensive tasks. Aside from that, it may be perfectly adequate for the job.<p>The only other factors would be if there were a language that offered features (or libraries) that you need.
niftichalmost 9 years ago
Does your product work? If yes, you&#x27;re not doing too bad.<p>Under what circumstances does your product break? Test this, don&#x27;t just speculate!<p>Are you finding that you&#x27;re having to jump through too many hoops to get the code to do what you want, and are wishful for a different language&#x2F;platform&#x2F;API? If you&#x27;re fighting against your current tools, investigate refactoring, or replacing small components to see if they help.<p>Never throw anything that worked away! Version control, version control!
bigiainalmost 9 years ago
Depending on what assurances and promises you&#x27;ve made to other people about the robustness of your current solution, I reckon there&#x27;s a high likelyhood you haven&#x27;t &quot;screwed up&quot; at all. You&#x27;ve investigated your problem space, found some workable solutions, and got some proof-of-concept code running.<p>Writing something in a language you know is always better than not writing anything in a &quot;better&quot; language that you don&#x27;t know. For _you_, Javascript was the right language, at least for your initial explorations. You&#x27;re now a long way ahead of where you started, with new-found knowledge about some of the edge case problems and implementation details that you never had when you started.<p>Javascript async is difficult (I&#x27;d hesitate to jump all the way to &quot;retarded&quot;), mostly because async itself is difficult (though, to be honest, a lot of people consider JS to have implemented it badly as well, but that&#x27;s kind of secondary).<p>Having said that, there are going to be similar downsides to doing it in Python, Ruby, Haskel, Erlang, Go, Groovy, Befunge, Brainfuck, Assembler, Perl - that&#x27;s why there isn&#x27;t just one language, they all have different compromises, different learning curves, different support communities, different library availability and robustness, different expertise availability...<p>Try not to ask &quot;how much have I screwed up?&quot;, but instead ask &quot;what has this version taught me?&quot;, and &quot;which of the decisions I made would I have made differently if I knew at the beginning what I knew now?&quot;<p>Bottom line - your project _can_ be written in Javascrip. It&#x27;s almost certainly significantly less complex than Gmail or Googlemaps. It just might take some very senior level Javascript developers to get there.<p>Don&#x27;t think that jumping from college-project-level Javascript to I&#x27;ve-read-a-bit-about-it-level Ruby or Python is going to magically make the hard things easy though.
评论 #12055924 未加载
bwackwatalmost 9 years ago
If you have a working implementation and have yielded moderate results, then you have earned &quot;success&quot;.<p>In this case I get the feeling you <i>want</i> to use a different set of technologies to either improve performance or ease extensibility.<p>How much did you screw up? 7&#x2F;10. You have found that node.js and mongodb are not the most mature technologies and would likely find python + sql (mssql, sqlite, postgresql, whatever) ... nicer.
asimuvPRalmost 9 years ago
You only know how to build a good program after you&#x27;ve built it. Go back and re do it. Repeat as much as possible.