TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

R3, a map-reduce engine with Python and Redis

76 pointsby mricordeaualmost 13 years ago

9 comments

cypherpunks01almost 13 years ago
Do the map-reduce results get put back into redis? I always worry about OOM problems when I'm putting a somewhat unbounded set of things into redis.
评论 #4303935 未加载
binarycrusaderalmost 13 years ago
Neat; but seems to missing copyright notices and an explicit license, which means no one can actually use it or redistribute it with their application.
评论 #4299373 未加载
评论 #4308550 未加载
fsaintjacquesalmost 13 years ago
Can you horizontaly scale the redis backend or it supports only one instance?<p>Why restrain to sequential reducers when you can parallelize with partitions and sorting?
评论 #4308555 未加载
grantjgordonalmost 13 years ago
Anyone have some insight into situations where running map reduce on redis makes more sense than other software like the traditional hadoop?
评论 #4298580 未加载
评论 #4300101 未加载
ChristianMarksalmost 13 years ago
I can think of one case where a redis dictionary is used to represent a tree, and reductions are needed over a subtree. Calculations on river networks are like this. You might want to use redis instead of a cPickled dictionary, and you might not want the overhead of a full Hadoop.
评论 #4300014 未加载
iandanforthalmost 13 years ago
"Getting one up in your system is beyond the scope of this document."<p>- 67 characters<p>brew install redis<p>redis-server<p>- 31 characters
评论 #4299594 未加载
评论 #4299236 未加载
brandynwhitealmost 13 years ago
This is pretty interesting, I have a related project (plug, hadoopy.com). The way I went about this (in an experimental branch) is to use Celery running on Redis.
dchichkovalmost 13 years ago
Can multiple users run tasks simultaneously? Can they set task priorities?
评论 #4308560 未加载
velodromealmost 13 years ago
This looks like an interesting project.<p>Is there something like this for php?
评论 #4298718 未加载