Organizing threads by albedo

[Nords]

Andy, BMJ, strictly for the benefits of reciprocated diatribes spirited debate, here's a new article from Philip Greenspun (experienced ER & MIT computer geek) & Julie Melton (a Harvard computer geek) on how to design a discussion board:

Solar Magnitude Forum

Apparently it can be implemented, perhaps painfully & slowly, in SQL...

[REWahoo]

Nords, did you mis-title this thread? I read your link and failed to see any connection to the inner rind of citrus fruit.*



*No smileys were harmed in the creation of this post.

[BigMoneyJim]

Interesting read.

My current hammer that makes every problem look like a nail is Bayesian filtering (most famously used for spam filtering). The implementations I'm looking at statistically decide whether an email or post falls into the category of "spam" or "not spam". But it takes someone to train the filter by submitting examples of each and correcting it when it's wrong. I've been thinking of different uses for it like "allowed poster" versus "poster non grata" and have wondered if it can identify hot button (for some people) posts like "offensive language" vs. "family language" or "attack" vs "not attack".

I wonder if it could be trained for "my interests" versus "not my interests" to obtain a result like what is described in the article. But to have this customized for each reader there would have to be a distinct statistical database for every reader. Also everyone would have to train their own filters. That sounds like a lot of trouble both server-side and for most readers.

Something I've thought about is to have multiple filters, but grouped instead of individualized. Suppose a poster here whose interest is in individual stock picking trains a filter for his interests and other users can use his filter if they share the same interests. Another user is interested in funds-of-funds and discussion of comparison, and others with similar interests. So in effect you're borrowing someone else's tunnel vision. These examples are much more narrow than I expect in reality, though, and such narrowness is better handled through traditional categorization, anyway.

Incidentally, I don't have the same powers at Andy's servers that I did at dory36's server (for several good reasons for both Andy and me), and I don't expect to be tweaking the site features here like I did in 2005 and 2006. (No more Days of Jennifer ) I'm just a simple moderator now, and a pretty lazy one at that.

I have my own sites, and these are ideas I've been kicking around implementing for them. My sites aren't particularly interesting at the moment, but I've recently started toying with them again. At the moment I think I've decided to ditch pre-built software and scratch-build the software myself as it's becoming troublesome to customize pre-built software the way I want while still keeping up with software updates that break my customizations.

[BigMoneyJim]

Greenspun is correct in that this can't be handled by a simple database query. If my Bayesian filtering idea wouldn't work then I imagine something that works more like a search engine would be used except that the searching would be automatic.

Another similar feature is the "tagging", "folksonomy" or reader-implemented categorization that's all the rage these days. But it's more of an emergent categorization scheme than a personalized one. Still it might be neat to have a "mortgage" tag that brings up all the mortgage-related threads (as identified by readers), for example.

Slashdot has such a tagging scheme implemented. Look for "(tagging beta)" under each article summary; to the left are tags that readers have assigned to the article. You can click various tags to see other articles tagged with that tag.

The folksonomy/tagging concept is popular enough that there might be a vBulletin plugin for it.

Hey Andy, I just had an idea: implement tagging, but make the tags span all your boards. Talk about synergy! I know we had a thread here recently about the royals; I expect some financial or retirement stuff pops up at Cruisers Forum occasionally. While we're on the topic of crossover participation, it would also be cool if users of one forum could post at another without creating a new account. Perhaps I could post at Cruisers Forum and show up as "BigMoneyJim (ERForum)". (I originally typed it to look like an email address with the domain name then decided that's a bad idea for several reasons.)

[cute fuzzy bunny]

I did quite a bit with Bayesian algorithms some years ago. Turns out they might be great ways to drive a demand for more cores in a processor...

[BigMoneyJim]

Guess I'll have to see what AMD and IBM are offering in the way of multi-core, then. And is somebody still building SPARCs?

[cute fuzzy bunny]

Watch out dude...I hear some of those AMD slabs dont even work with an external monitor...

Fujitsu still makes a sparc64 chip but I'd bet they dont sell many. Nice chip in its day for its purpose.

[Andy R]

Thanks for posting the link, I love reading this stuff. I have to continue my work on moving data centers but will revisit this thread ASAP.

Here is a post I made a couple weeks ago here to get some feedback on organizing E-R.org better.
http://www.early-retirement.org/foru...ing-28714.html

I can also put in text based tagging (ie. not pre-defined categories) but then we get into issues with dirty data. I would prefer to go through the exercises of determining the proper taxonomy for E-R.org (which can remain a work in progress) and then put in an explicit tagging system (drop down lists, check boxes, etc) so that we can keep things clean.

I think an important goal should be to understand the different users we have an create the best interface. Not everyone does things the same so having multiple views to the same data is smart.

BMJ - I love the idea of tagging across all the sites.

FYI - Part of my vision for Social Knowledge is developing an entity relationship database where data specific to each vertical niche topic will be kept. Think IMDB but for other niches with interrelated data. I can't disclose too much at this moment but this database is a huge component of my vision for Social Knowledge. If your intrigued and want to learn more, send me a PM and we can talk.

[twaddle]

Forget taxonomies and tags. Doesn't anybody here use eBay? It's all about search. If you want to cross-pollinate, then add a multi-forum search capability and focus on improved search speed and quality.

[BigMoneyJim]

Interesting: Google Gears is a browser-implemented database engine for local storage.

Basically the end user downloads the plugin and a web application has local, persistent and relational storage. I can think of lots of uses for this...

Imagine FireCalc storing multiple runs of data on your computer rather than having to recalculate and redisplay every tweak. You would even be able to view your previous results if your internet connection or FireCalc were down.

I post it here because it may be useful for implementing the ideas discussed above. Offloading some work to the client can result in greater personalization of data without overloading the server.

[Andy R]

^ BMJ - Google Gears is a cool concept and long time overdue. I will have to start slow (gotta walk before we can run) but you are right, that would be an amazing way to push data to the client and hit the server less. Unfortunately I am a long ways away from that point right now. Tuesday is my first meeting on database modeling for the entity relationship database and we will start with Fashion (SK's biggest site). The plan is to map the who, what, when, where & why of fashion. This will be like IMDB of fashion and the start of a larger relational database on all kinds of topics.

If you want to look at a really cool project, I think we should toss around the idea of developing a predictive market system for E-R.org to foster a marketplace for buying and selling the probability of events that effect portfolios.

Here is a very cool application that was put in place at Google using a system like this: Putting Group Wisdom to Work

I worked a lot with Monte Carlo simulations (multiple regression analysis) of portfolios at my last job at RIA firm. It would be extremely interesting to tie historical statistical information (per asset class) to a predictive marketplace for events and develop correlations between events to asset classes. Not a little project but sounds like fun...