Vanilla 1.1.9 is a product of Lussumo. More Information: Documentation, Community Support.
I've just turned on http://alpha.mathoverflow.net/. As the name suggests, this is an extremely preliminary reimplementation of the software MathOverflow runs on. It's currently running against a database dump from http://faketestsite.stackexchange.com/ rather than the real data from MathOverflow.
It's very possible that it will go back offline unexpectedly --- it's kindly hosted on the machine that runs the Knot Atlas, a joint project of mine with Dror Bar-Natan, and if alpha starts misbehaving and clobbering that machine I'll just switch it off again.
It took a few days to get the code to the point that you can see, but it's very far from what we'd eventually need, and I can't do the rest myself. (I need to do some mathematics every so often!) This post is to explain what we have now, and give a very brief tutorial on all the moving pieces.
At some point we actually need to think about whether we want to go down this road in the end. For now, I'd just like to see if there's interest from other members of the community in pushing this further, and to work on it at a low priority. I'm going to ignore all these higher level decisions in this post, and just talk about the technical details.
You can get a local copy of the software using mercurial (tutorial tutorial), by typing
hg clone https://tqft.net/hg/mathoverflow
(There's an alternative copy of this repository at http://code.google.com/p/mathoverflow/source/list, but for now it's not necessarily "the bleeding edge".) This repository also contains various other bits of code that Anton and I use to maintain MathOverflow; everything you need for alpha is in the alpha/ subdirectory.
To compile and run a local copy, you need to have maven (>= version 2.2) installed. To get a local copy running at http://localhost:8080/, type
mvn jetty:run
If you modify the source code and recompile (with mvn compile) the local webserver should automatically restart with your changes. The actual database resides in an SQL server at my web hosting, so even for the local copy you need an internet connection for anything to work.
Alpha itself is written in a modern programming language called scala, and uses the Lift web framework. Unfortunately both of these have pretty steep learning curves, which is going to be a really significant problem for getting people involved! On the other hand, both are pretty fantastic. Scala is essentially "what java could have been": it's statically typed with lots of clever compiler inference, allows imperative style programming but encourages functional programming, and is both concise and expressive. Lift is not your everyday web framework, and web frameworks generally are not something I have any expertise in. I've been very happy to find that implementing fancy tricks via AJAX and Comet (i.e. dynamic web pages) is straightforward and intuitive, and doesn't even involve writing any Javascript by hand (essentially, Lift compiles scala to javascript as needed).
Before we get onto the codebase, there's a issue tracker for alpha. Please feel free to add issues, but for now please restrain yourself -- obviously there's a huge gap between alpha and the real thing. Let's only add issues detailing discrepancies between alpha and the "read-only" aspects of MathOverflow (e.g. don't worry about voting, answering, login, moderating, etc, for now) and even then, don't bother with issues for minor details that look like they'll get fixed in the course of other essential work.
If you actually want to look at the code, I highly recommend installing Eclipse, an IDE. You'll also need to install several Eclipse packages (go to "Help" -> "Install New Software", and enter the following update sites):
Finally, the code! (btw --- if you're actually following along the instructions, I'm very happy to help you get things set up: send me an email)
There are three main components:
Don't worry too much about the .db.* packages. It's pretty simple: it uses the Circumflex framework for talking to the database, and mostly just contains the definitions modeling the underling database tables. It's important to at least look at these to understand which data is stored where.
The .data package contain various representations of Posts,
Comments, Users, and so on. Again, mostly this is not too exciting
--- it mostly exists to disintermediate the presentation and database
layers, so that when you're thinking about actually writing webpages
you don't need to be thinking about SQL statements at the same time
(also so that we can switch to different mechanisms for talking to the
database easily). There's a bit more "modeling" work to be done here;
more esoteric things like votes to close and badges just get ignored
for now.
The .lift.* packages contain most of the fun, and most of the work to be done! To get a sense of what exists in here, you should first poke around http://alpha.mathoverflow.net/. Everything is generated by code in either the .lift.view package or the .lift.widgets package. "Views" render entire pages, "widgets" render, well, widgets (things like a post signature, a list of tags, a set of sort order tabs, etc). Now, somewhat scarily, all of the actual HTML eventually sent to the client is contained inside various Scala source files. Why is this scary? Well, web people really like to have their "presentation" and "business logic" nicely separated (essentially, so that designers and programmers don't have to talk to each other much), and HTML inside source files really breaks this goal. It also means that for a given problem that you see in the rendered HTML, it can be a bit of a maze finding the actual point in the source code it was produced. I should say this isn't Lift's fault: as a framework, it's pretty flexible, and gives you the option of going to either extreme in this regard. (In fact, two days ago alpha really was at the other extreme, where all of the rendered HTML was actually in template files, and the code rewrote snippets of these templates to insert the appropriate content.) Changing this, or using a hodge podge of both setups, is a design decision we can revise later. For now, with a single omnipotent developer, it's hasn't been much of a problem. The thing that makes this design choice much less scary is that Scala handles XML really really nicely --- you can just start typing XML anywhere in code, and break back out into Scala with curly braces. Thus the code in the .lift.views package mostly looks like a single chunk of HTML, with occasional short sections of interspersed Scala to fill in a text field or embed a widget.
Now, for the most part alpha just produces static HTML. Eventually it will have to have all sorts of interactive features, like voting, answering, commenting, etc. Happily Lift makes this sort of programming pretty painless, but I won't talk about this here until there's someone around to listen. :-) The only example of dynamic content so far is the user browser at http://alpha.mathoverflow.net/users/, which might not even look interactive now because there aren't enough faketestsite users to need multiple pages. But take my word that it works nicely!
Okay.... go have a look, and let me know how it goes.
@Ryan, essentially, yes.
I get a lot of unsupported node 4 errors.
Very cool! This is definitely a step in the right direction.
I get an error:
Proxy Error
The proxy server received an invalid response from an upstream server. The proxy server could not handle the request GET /.
Reason: Error reading from remote server
@Ben,
It's back up, who knows what went wrong.
@bbischof,
If you or anyone else sees these "unsupported node 4 errors" again, could you copy and paste them to me via email, as I've know what what's being referred to.
@Gerry,
:-) It's actually pretty amazing these days the tools available. Standing on shoulders of giants, etc, very much applies to this stuff as well.
Way out of my depth here, but are there any licensing problems with mimicking the SE software?
Might there be any issues with SE doing something vindictive and shutting down MO before this project is done?
So, wait, are you re-engineering the SO site from scratch? Hasn't OSQA already done that? Is there some reason we don't want to use them?
Maybe Scott thinks he can do it better with his preferred tools. Recall that OSQA is written in python, while Scott is using a whole bunch of stuff I've never even heard of!
@Henry,
it's complicated. Certainly alpha at the moment does two evil things: it copies the CSS and sprites (the little images for voting, favorites, etc) directly from Stack Exchange. This definitely needs to be replaced at some point.
On the wider issue of copying the general layout, it's unclear. Certainly if they were feeling mean they'd have sufficient legal standing that we couldn't afford to ignore nasty letters. On the other hand, they seem to be perfectly comfortable with what the existing competitors have done. There's even some blog post by Joel Spolsky, which maybe Anton knows how to find, specifically addressing this issue.
@Ben,
yes, this is starting from scratch.
I have to admit I haven't looked at all at the OSQA codebase. I did get the feeling when we first talked to the OSQA folks that in various ways we weren't seeing eye to eye. This made me doubt we could ever persuade them to push OSQA in the directions we wanted. Moreover, they had ideas about future directions (social networking style features) that seemed pretty dubious to me.
Thus, my guess would be that at best we'd be forking OSQA at some point, and going our own way. That might well be fine, and we'd certainly have a huge headstart over alpha. On the other hand, starting with other people's half-done work is a huge pain in the butt. Personally I'd prefer to deal with something we could be more perfectionist about.
The toolchain I'm using, scala+maven+lift is a lot to digest. It's not impossible that me and Dave Penneys, a graduate student at Berkeley, are the only mathematicians who've programmed in scala. On the other hand, everyone can write python :-)
Scott, I wondered as well about why we're not just forking OSQA or some such engine. After all, forking an open source project seems a whole lot better than rewriting it from scratch. I agree that wading through someone else's code is a nightmare, but it might be (I haven't looked) that most of the differences between what MO wants and what OSQA offers is modular: namely, that the core is fine but the modules on top need tweaking. That's essentially what I've done with the vanilla forum software and the nForum: the core is sound, and is far better than I could ever design, but it needed a little tweaking to make it suit my needs.
I've not programmed in python or scala, so can't offer to help out in either way there. However, I do know that itex2MML works fine with python and I don't know about scala, but I could find out if there's any chance that with this change I could reopen the debate as to server-side mathematics conversion as opposed to client-side.
@Scott On each page I get the error as soon as it loads: "unsupported node -4" that is all the error says.
If I enter a question, and then click on the banner I get a text page that says "Exception occured while processing /
Message: java.util.NoSuchElementException: None.get
scala.None$.get(Option.scala:262)
scala.None$.get(Option.scala:260)
ru.circumflex.orm.ValueHolder.apply(common.scala:147)
net.mathoverflow.data.User$$anon$1$$anonfun$2.apply(User.scala:80)
net.mathoverflow.data.User$$anon$1$$anonfun$2.apply(User.scala:80)
scala.Option.getOrElse(Option.scala:104)
net.mathoverflow.data.User$$anon$1.
where ... means about another 100 lines.
Oops, thanks bbischof, you're absolutely right. Hopefully it's working again now (is for me), and this time it's running on the December 1 public database dump from MathOverflow, rather that FakeTestSite.
Still unsupported nodes. Banner fixed, but the date links are broken on the questions. Also, Mod, delete and flag options are on every questions, but I cant click them. If I click edit, it leads to a broken link page. The link option just does nothing.
@bbischof, could you send me a screen shot of the "unsupported nodes" thing? It sounds like a problem, rather than just something not yet implemented, and I don't really understand what you're describing.
I just made some upgrades to alpha.mathoverflow.net. Amongst other things:
It's also much faster for me than it was before.
I can't find any revision histories. The posts don't have a "last edited XX TIME ago" link, and the URL I expect to work (/revisions/POSTID/list) doesn't work.
Hmm, here's an example revision page: http://alpha.mathoverflow.net/revisions/49426/list
Some of the revision pages generate error messages, for example anything that became community wiki at some point, but otherwise they seem okay. You're right that the post signatures aren't right, and "last edited ... ago" links are missing. However all the questions and answers should have a linked date, e.g. "asked Dec 14", and that date goes to the revision page.
Nice project! But I have not yet had the opportunity to view it. All I get every time I try is this:
"Service Temporarily Unavailable The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later. Apache/2.2.8 (Ubuntu) DAV/2 SVN/1.4.6 PHP/5.2.4-2ubuntu5.12 with Suhosin-Patch mod_ssl/2.2.8 OpenSSL/0.9.8g Server at alpha.mathoverflow.net Port 80"
When is it available?
Sorry, it is indeed down. I expect that I'll next find time to work on this in mid-March. My main priority then will be to find somewhere stable to run it (previously it's been on the same server as http://katlas.org/, but not set up to restart if the machine reboots), and to organize and document the code sufficiently that other people can get involved.
@Scott, thanks for the response. @ Noah, I think so many are in favor. I, being an undergraduate in his final year in a country far from the Western or Eastern countries, do not know to whom my proposal would go. I care a lot about the moderators. Their selfless devotion to this site is laudable and indeed a noble deed. The best I can imagine is if some day some AI gets developed and gives relief to future moderators(being optimistic, to current ones, too).
@Mike,
there's no such list outside my head. It's possible to get a copy of the existing code, and indeed run a local copy of alpha (see above). However my available time for this is going to be best spent working privately on it, for at least a few more full days of work, than getting other people started. I very much look forward to the moment when that changes, however, and I'll keep you in mind!
1 to 29 of 29