[web-devel] Sql join feature in persistent
mxcantor at gmail.com
Tue Feb 22 12:50:48 CET 2011
Agree. As long as its done in the app-level, it doesn't affect scaling.
On Feb 22, 2011, at 7:19 PM, Антон Чешков wrote:
> We all want to achieve the same goal make data join in application layer.
> From my point of view the main problem is effective scale join algorithm.
> For example i have two datasets "A" and "B".
> I want join its by some rule and take only first "N" results. In this case i do not want serve all datasets entirely. I would like to make as little as possible.
> I think this algorithms are exist.
> 2011/2/22 Max Cantor <mxcantor at gmail.com>
> I strongly disagree.
> Because Persistent doesn't do joins, it happens to be a good candidate for a "web-scale" horizontally sharded solution, which I'll get to in a minute.
> Its perfectly easy to perform joins on the app-server in haskell code rather than relying on the database to do them. Facebook does a lot of this. We do this. Note that its much easier to replicate app-servers than DB servers.
> About making persistent-webscale, there are two features whose additions would basically accomplish this. Michael, I am thinking of offering a small bounty if you want to put them together.
> One is sharding. In the mkPersist function, if you could add a partition function which is somethng like: PersistEntity e => e -> SqlConnection and then alter the PersistBackend instance to respect that function, you get sharding almost for free. Its a fair bit more complicated than I am describing here but I believe its very possible. joins would make this inordinately more complicated.
> Also, integrated memcached. It would be great if you could annotate tables to store in memcached. when you set up your connection, optionally include a memcached connection. then persistent would check in each select, get, getBy call if the value is in memcached before going to the DB.
> I'm not very familiar with persistent internals so I could be wrong, but I think these shouldn't be too hard to add.
> On Feb 22, 2011, at 5:37 PM, Michael Snoyman wrote:
> > Hi James,
> > It's funny you mention this now. I recently got an email from Антон
> > Чешков about this very topic. Hopefully he can share the code he sent
> > me with the list. The one improvement that we're planning on making is
> > to write it using enumerators instead of lists so that the memory
> > requirements are lower.
> > For now, however, you might have noticed that the Haskellers code does
> > a number of "pseudo-joins" all over the place, which currently works
> > well enough. Антон's suggested code basically automates a lot of this
> > process.
> > Michael
> > On Tue, Feb 22, 2011 at 11:25 AM, James <cnjamesdeng at gmail.com> wrote:
> >> Hi all,
> >> I'm new to this mailing list. I did web development in python with
> >> Django framework before. But I really like the haskell language, so
> >> considering building my next site (a LBS-SN like service) in haskell. I
> >> learn snap for a while and think it's not so mature and powerful. After
> >> I learned the yesod docs in one day and read some yesod and
> >> haskellers.com codes in the following few days, I appreciate it and try
> >> to use it seriously.
> >> Now I need a database query across two table, so as to sql join feature.
> >> Currently, persistent doesn't provide this functionality. I think maybe
> >> adding a function to PersistBackend class, and implementing it in
> >> PersistBackend (SqlPersist m) may solve the problem. However, I am not
> >> sured. Will it be difficult and consistent to implement this feature?
> >> Thanks for any help.
> >> Sincerely,
> >> --
> >> James Deng
> >> department of computer science,
> >> school of information science & technology,
> >> Sun-yat-sen University, Guangzhou, China
> >> mailTo: cnJamesDeng at gmail.com
> >> homepage: http://cnjdeng.appspot.com
> >> _______________________________________________
> >> web-devel mailing list
> >> web-devel at haskell.org
> >> http://www.haskell.org/mailman/listinfo/web-devel
> > _______________________________________________
> > web-devel mailing list
> > web-devel at haskell.org
> > http://www.haskell.org/mailman/listinfo/web-devel
> web-devel mailing list
> web-devel at haskell.org
> Best regards,
> Cheshkov Anton
> Phone: +7 909 005 18 82
> Phone: +7 931 511 47 37
> Skype: cheshkov_anton
More information about the web-devel