tea.mathoverflow.net - Discussion Feed (MO public database dumps)

darijgrinberg comments on "MO public database dumps" (22074)

2013-05-09T18:17:41-07:00

About half a year has passed since the last one...

Anton Geraschenko comments on "MO public database dumps" (19672)

2012-08-13T16:36:29-07:00

@darijgrinberg: I will definitely generate a new public dump before the 2.0 update, and will maintain redundant copies of the full dump.

darijgrinberg comments on "MO public database dumps" (19650)

2012-08-13T01:30:56-07:00

How about a dump before the 2.0 update?

darijgrinberg comments on "MO public database dumps" (18889)

2012-04-12T14:40:13-07:00

Thanks for making it work again!

Anton Geraschenko comments on "MO public database dumps" (18885)

2012-04-12T08:31:28-07:00

Permissions issue resolved. The most recent dumps are now available.

Anton Geraschenko comments on "MO public database dumps" (18879)

2012-04-09T11:43:39-07:00

Sorry, my fault. Unfortunately, I have to do some things manually each time. For a while, it became overwhelming because the database export feature broke (it's working again now that MO is on a new server), so I had to contact SE by email to get the dumps. I still got the snapshots, but didn't clean them up for public consumption. I'll get the last few dumps up within the hour. [ok, maybe a bit more than an hour; my up-bandwidth isn't so great and the dumps are >100MB each]

Edit: hmmm ... I'm running into some permissions issues. I'm checking with Scott to see if he changed something.

darijgrinberg comments on "MO public database dumps" (18876)

2012-04-08T15:59:40-07:00

So what happened to the dumping? I thought there was some script doing that automatically, or does Anton have to tweak things every time?

Harald Hanche-Olsen comments on "MO public database dumps" (18545)

2012-02-21T07:42:29-08:00

@Anton: Too busy to make notes of new dumps here? These alerts are much appreciated, if only for avoiding this thread getting buried too deep. It's the place to go when looking for fresh dumps, after all. Oh, and the latest dump is dated 2012-01-03. Is the next one overdue?

Anton Geraschenko comments on "MO public database dumps" (15349)

2011-08-01T23:06:05-07:00

There's a new dump posted.

Anton Geraschenko comments on "MO public database dumps" (14889)

2011-07-07T11:34:33-07:00

There's a new dump posted.

Anton Geraschenko comments on "MO public database dumps" (14675)

2011-06-04T11:31:31-07:00

There's a new dump posted.

Anton Geraschenko comments on "MO public database dumps" (14437)

2011-05-10T01:57:44-07:00

Sorry for not posting a dump last month, and for being late this month. My excuse is that I've been writing my dissertation.

There's a new dump posted.

Anton Geraschenko comments on "MO public database dumps" (13631)

2011-03-13T12:58:27-07:00

There's a new dump posted.

Anton Geraschenko comments on "MO public database dumps" (13079)

2011-02-04T08:42:45-08:00

There's a new dump posted.

Anton Geraschenko comments on "MO public database dumps" (12308)

2011-01-02T00:07:03-08:00

There's a new dump for the new year.

There are a couple of changes (mostly discussed in the few posts above this one). The biggest one is that post histories are now included! See the readme for details about what all the fields mean.

Anton Geraschenko comments on "MO public database dumps" (11806)

2010-12-13T01:04:59-08:00

Okay, I've updated the public dump script to include the RevisionGUID for each revision (that's right, the dumps will contain revision histories starting 2011) and GravatarHash for each user.

LastEditorDisplayName isn't in the non-public table unless the user has been deleted (and so LastEditorUserId doesn't exist). Looking the the SO public dumps (at least the Stack Apps dump, since that's the smallest one to deal with), it looks like LastEditorDisplayName is always given, but is always empty!

Scott Morrison comments on "MO public database dumps" (11799)

2010-12-12T17:56:39-08:00

Re: my post above on Dec 4th.

I see you've removed the reference to LastEditorDisplayName in the README.txt. Can't we instead include all of LastActivityDate, LastActivityUserId and LastActivityDisplayName, in the public dumps? Surely this is public data too!

Scott Morrison comments on "MO public database dumps" (11798)

2010-12-12T17:53:50-08:00

Two requests for the public dumps:

Include the GUID for each revision, in posthistory.xml. This is public information, contained in the URL for "view source" on the list of revisions.
Include the Gravatar hash: take the user's email address or last login IP address, as a string, and compute the MD5 hash.

Scott Morrison comments on "MO public database dumps" (11257)

2010-12-04T12:24:58-08:00

The README.txt for the dumps says that posts.xml should contain a field "LastEditorDisplayName". It doesn't seem to.

Anton Geraschenko comments on "MO public database dumps" (11178)

2010-12-01T15:03:38-08:00

I've just posted the December dump.

Anton Geraschenko comments on "MO public database dumps" (10153)

2010-11-03T16:45:07-07:00

I've just posted the November dump.

Anton Geraschenko comments on "MO public database dumps" (9593)

2010-10-14T22:29:55-07:00

No reason to censor; it just takes a bit of work to include. I've been meaning to do it for a while, but I haven't gotten around to it yet. When I originally wrote the code to generate public dumps, I modeled it after the SO public dumps, which don't include post histories for some reason.

There are two tables in the database. One contains the posts in their current state in html (for fast serving). The other contains an entry for every edit, retag, or other action that can be taken on a post; notably, it contains the markdown source. It shouldn't be too bad to include another file, posthistories.xml, in the public dump. It would make the dump perhaps 50% larger.

Scott Morrison comments on "MO public database dumps" (9592)

2010-10-14T20:40:52-07:00

Anton didn't announce it, but he put up a dump for October, too.

A question about the database dumps -- they don't include the edit history of posts. I assume, @Anton, that you have these in the full dump? Was there some reason to censor these?

Bill Dubuque comments on "MO public database dumps" (7985)

2010-08-03T19:10:45-07:00

Thanks much Anton.

Anton Geraschenko comments on "MO public database dumps" (7984)

2010-08-03T19:04:57-07:00

I've just posted a fresh database dump. By popular demand (somebody asked for it), I've included the hour each vote was cast. Before it only included the day. This should be enough to do some rough analysis of voting habits. If you want aggregate statistics finer than that, you'll have to ask me to look at the full dump.

Kevin Buzzard comments on "MO public database dumps" (6732)

2010-07-06T23:44:08-07:00

Quick and hopefully easy question, I thought that a recent question on MO was a duplicate and I couldn't find it using MO search, so I turned to the dumps and I found what I was looking for using grep. I now have a comment from comments.xml [i.e. a line in that file] and I want to locate this within the MO site. The comments.xml line starts "row Id="6664" PostId="5703"". Should this be the information I need to locate the comment within the site? I also know who posted the comment and when they posted it, but this doesn't seem to be much help in actually locating the comment on the site.

EDIT: I have solved this myself. The trick I found is to go to http://mathoverflow.net/questions/5703 and you get redirected to the right place.

Andrey Rekalo comments on "MO public database dumps" (6681)

2010-07-03T13:52:03-07:00

Thank you. This is very useful.

Anton Geraschenko comments on "MO public database dumps" (6632)

2010-07-01T16:54:59-07:00

Early 4th of July present: a fresh database dump.

Scott Morrison comments on "MO public database dumps" (5697)

2010-06-01T08:55:39-07:00

Oh dear, is it June already? :-)

Anton Geraschenko comments on "MO public database dumps" (5695)

2010-06-01T08:32:32-07:00

Happy June. I just posted a fresh database dump.

Scott Morrison comments on "MO public database dumps" (5231)

2010-04-29T15:37:02-07:00

Fixed. Copy and paste error, I presume.

Harald Hanche-Olsen comments on "MO public database dumps" (5228)

2010-04-29T15:26:58-07:00

Why does the zip file have the same filename as the previous dump?

Anton Geraschenko comments on "MO public database dumps" (5216)

2010-04-29T12:18:24-07:00

I just posted a fresh database dump.

Scott Morrison comments on "MO public database dumps" (4441)

2010-04-05T10:32:29-07:00

Okay, I agree that the benefits outweigh the slight, arguable, violation of the "only public data" rule.

Anton Geraschenko comments on "MO public database dumps" (4326)

2010-04-02T14:11:08-07:00

Yes, it does mean that there is some information in the dump which is not available through the website: votes to close have user ids even if they are not "effective". Non-effective votes to close occur in two ways: either not enough votes to close have accumulated on a question (e.g. there are four such votes right now), or the vote to close has expired (votes to close only have a lifespan of four days).

When you vote to close (or reopen), you're volunteering to associate your name to that vote (after all, your name is displayed once the question is actually closed/reopened), so I don't feel like anybody can reasonably object that they meant to keep their identity private when voting to close. The main reason I think it's good to not display who has voted to close a question which is not yet closed is that people vote differently when they know who is "on their side". Not displaying who has voted to close is a way of getting people to take personal responsibility for their vote to close. The dump contains so little information that undermines this purpose that I don't think it's a problem. I'm happy to change it back if somebody has a good reason to do so.

Scott Morrison comments on "MO public database dumps" (4325)

2010-04-02T11:13:24-07:00

Just a sec: doesn't this make "votes to close" publicly identifiable before the question actually gets closed?

Anton Geraschenko comments on "MO public database dumps" (4321)

2010-04-01T22:09:54-07:00

The April database dump is now available, fresh off the server. I've corrected/changed a couple of things in the script I use to produce the public dump. Specifically, vote tallies are now included in the users.xml file (in the last dump, I claimed in the readme that they were included, but they weren't actually). Also the votes.xml file now includes the UserId when somebody votes to close or reopen a question.

Anton Geraschenko comments on "MO public database dumps" (3654)

2010-03-04T16:32:01-08:00

You can now access the dump (or any part of it) at dumps.mathoverflow.net.

Harry Gindi comments on "MO public database dumps" (3643)

2010-03-04T00:11:21-08:00

That was Anton, before I suggested putting it on a free hosting site.

Scott Morrison comments on "MO public database dumps" (3642)

2010-03-04T00:05:52-08:00

Also, someone has already put up a torrent, for example at http://thepiratebay.org/torrent/5408662/.

Scott Morrison comments on "MO public database dumps" (3641)

2010-03-03T23:51:35-08:00

It's now available at http://dumps.tqft.net/.

Soon after Anton reads this, you can access dumps at http://dumps.mathoverflow.net/. (Anton, do the usual! I'll give you shell access as well.)

Anton Geraschenko comments on "MO public database dumps" (3638)

2010-03-03T23:20:04-08:00

You're right. I'll just do that. Edit: done.

Harry Gindi comments on "MO public database dumps" (3636)

2010-03-03T23:10:44-08:00

=(. The database dump is 9.6 MB. Why don't you rapidshare/megaupload/etc it?

Anton Geraschenko comments on "MO public database dumps" (3634)

2010-03-03T23:02:55-08:00

Here's the first public dump of the database. It's from earlier today.

http://dumps.mathoverflow.net/, or
http://ifile.it/soyqa09/MOdump20100303.zip