I am currently in the process of writing up a paper based on the survey of motivations to use MathOverflow. In the mean time I wanted to share with you the results of another study I did on MO as well as some basic demographic data from the survey.

http://www.utpsyc.org/MathOverflowSurvey/MathOverflowStudies.php

Thanks again to everyone who participated.

Thanks for the update!

@Ben: my interpretation of Angelo's comment is quite different---we'll have to let him clarify, but my guess is that his point is that if you take someone like Andrew Wiles, and count his published papers, you'll get something like 20. On the other hand, I know at least one number theorist who has published over 100 papers. Am I to conclude that these other number theorists are "five times better than Wiles"?

@Yla,

I'd say the number of published papers is such a wildly imperfect measure of offline reputation that it is irresponsible to use the phrase "offline reputation" to refer to this number.

@Yla: While I would not go far as to call the phrase "offline reputation" irresponsible, I'd argue that the burden of defending metrics in your survey falls on you, not on Scott Morrison. Indeed, I would expect such a defense to be part of any publication on this issue.

Regardless, that's an interesting survey! Though I am curious about what status those who claimed to be "status seeking" on MO hope to achieve.

@DL I agree that it is my responsibility to defend my measure. Here is why I think it is a good enough measure for the purposes of this study.

I think there is a misunderstanding about what kinds of MO users we are trying to distinguish between. "Number of published papers" is not an attempt to tell us whether Andrew Wiles or Grigori Perelman is the more accomplished mathematician. Professional mathematicians might be interested in such a thing, but this study is interested in the attributes of the bulk of the users of the site, not the upper tail.

I hope we can all agree that there is a clear difference in the amount of offline reputation that an undergraduate student has, and that a full professor has, and that the number of papers published will show that. In fact, I think it's likely that we could run the same analysis with offline reputation being a binary measure of whether or not the user had any published papers, and get similar results.

As far as "irresponsibility", it is common practice within psychology to try to measure a (ill defined and difficult to measure) feature of the real world by making an operational definition, and then naming the measure by the feature of the real world. Others can then argue about how good that operational definition is, but nobody thinks or claims that an operational definition completely captures the real world feature, or that it should be interpreted as the real world feature.

In the context of communities of user generated content this is the first paper with any kind of measure of offline reputation. Perhaps it only explains 10% of the "Platonic ideal offline reputation", but within my field it is standard to call it "offline reputation" even in this case. Apologies for any misunderstandings. I am open to any suggestions of other ways to measure offline reputation.

@Ryan Budney "offline reputation" is something people care about in the context of other communities (e.g. Yahoo! Answers, Wikipedia). "Number of papers" is not generalizable to these other contexts so it is less interesting within my field.

@Yla: I think people's concern here (as well as mine) is the following: the term "offline reputation" is evocative enough to connote a certain meaning: namely the degree to which a certain MO user is esteemed by the mathematical community. But in your work, you do not give any precise (or imprecise!) definition of it other than equating it with the number of papers published. This does seem problematic.

"I hope we can all agree that there is a clear difference in the amount of offline reputation that an undergraduate student has, and that a full professor has, and that the number of papers published will show that."

No, not yet. First we need to agree on what offline reputation means. If it actually means "number of published papers", yes, it will show that. But as I said above, you seem to be relying on some more vague idea of what the term means. One can find undergraduates with more published papers than full professors. For instance, Sophie Morel is a full professor at Harvard University and has, according to MathSciNet, one published paper. So no, your implicit equation is really not so clear.

"In fact, I think it's likely that we could run the same analysis with offline reputation being a binary measure of whether or not the user had any published papers, and get similar results."

Then perhaps you should try it and see.

"As far as "irresponsibility", it is common practice within psychology to try to measure a (ill defined and difficult to measure) feature of the real world by making an operational definition, and then naming the measure by the feature of the real world. Others can then argue about how good that operational definition is, but nobody thinks or claims that an operational definition completely captures the real world feature, or that it should be interpreted as the real world feature."

The word "irresponsibility" is a little bit strong here. But I don't entirely agree with what you say: that which is ill-defined deserves more efforts at attempted definitions and descriptions, not less. Similarly with things which are difficult to measure. In other words, it seems professional to explicitly address and argue for your equation of "offline reputation" and number of published papers. Moreover, I think the portion of your paper when you set off a red boxed number one and write "Offline reputation: Number of published math papers" is definitely a claim that your operational definition captures the real world feature.

I do think it is facile to present "measurements" without carefully addressing the extent to which the quantity that you're recording is actually a measurement of your phenomenon. To say "others can then argue" about it is kind of a copout, because that's not how things work in real life or even in academic life: academics have been told that "they can argue about" the actual meaning of student evaluation scores, impact factors and NRC rankings. Well, anyone can argue about anything but that doesn't change the fact that these statistics are now being used in making the most important decisions: hiring, promotion, tenure, funding, and so forth.

"Perhaps it only explains 10% of the "Platonic ideal offline reputation", but within my field it is standard to call it "offline reputation" even in this case."

Again, you seem to be presuming (perhaps facetiously, but that's kind of the point) that there is some Platonic ideal of offline reputation. What does this term actually mean to you? Moreover, in what academic field does the "offline reputation" have a meaning so standard that it appears in papers without definition or question? [I would be interested to see references.] Could this term really be more than a few years old? You're truly okay with reporting a measurement that might explain only 10% (or less?) of what it's desired to measure? Shouldn't you at least try to do better?

Here is a suggestion: why don't you measure offline reputation by a survey? But be sure to tell the surveyants what the term means!

I would not have dreamed of using the term "irresponsible". However, I agree with Kevin: all that the number of your papers measures is how many papers you have published. This has, I suppose, a positive, but very weak, correlation with your reputation as a mathematician. Indeed, publishing too much can hurt your reputation as much as publishing too little. To me, "At most moderate correlations between offline and online reputation" sounds very different from "At most moderate correlations between number of published papers and online reputation". The latter is, at the very least, far less surprising.

Now, of course, coming with a better measure is hard; but I hope I don't sound snotty if I say that if something can not be reasonably done within the constraints of a study (in this case, measuring offline reputation), perhaps should not be done.

Number of papers clearly is measuring something, because it predicts some amount of the ratings of submissions on MathOverflow.

Sorry to be pugnacious about this, but why then don't you call number of published papers "online reputation", rather than "offline reputation"? It seems you have a better case for this misnomer than the one you're using!

Of course this is being a devil's advocate, it would be silly to actually call number of published papers "online reputation". It's just that it seems to me that calling it "offline reputation" is equally inappropriate as it would be to call it all sorts of other things. Surely there are some criteria for deciding when "operationalization" has gone too far? (I admit of course, that I am totally ignorant of psychologists' practice here.)

@Yla,

responding to your earlier question

Also, do you think number of published math papers is a poorer measure of offline reputation than MO Points is a measure of online reputation?

could I first ask for clarification of what you mean by "online reputation" and "offline reputation" in the first place? I'm not really sure how I would try to define these separately. Perhaps we could try to do this as the respect held for someone on account of the actions performed in either an "offline" or "online" context. Thus while I'm deeply impressed by, say, T, as a mathematician, in my regard he has accrued some "offline" reputation when I've seen him give excellent public lectures, and some "online" reputation when he's posted great articles on his blog. But if I read one of his papers via the arxiv, and like it, does that contribute to his "online" or "offline" reputation? And if a trusted colleague tells me via email "Wow, T's latest result is fabulous", which is that?

I suspect that with a definition of online and offline reputations along these lines, I would argue that MO reputation is an extremely good proxy for "online reputation", but merely because the main other available source of it is writing blog posts, which relatively very few mathematicians do.

"Number of papers clearly is measuring something, because it predicts some amount of the ratings of submissions on MathOverflow."

Yes, number of papers is measuring something: the number of papers. You have collected this data and analyzed its correlation with ratings of MO submissions: sounds good. This is what has actually been done. I don't see the added value in equating "number of papers" with "offline reputation", as neither a theoretical framework for the latter is being presented nor an argument for the validity of the former as a measurement for the latter. You could explain that you chose to measure and analyze the number of papers out of a feeling that is capturing something about the status of the participants in the mathematical community, but that's about it.

You mention the problem that offline reputation is a concept which is being used in more general contexts where "number of papers" doesn't make sense. I understand and respect that concern, but I don't see how that just picking something and calling it offline reputation contributes to the solution.

Dear DL,

Thanks for making this comment. In particular, I support the sentiment of your third paragraph.

Regards,

Matthew

I also strongly agree with DL.

Point taken! :-)

Dear Ben,

I think that Yla already explained that the desired variable is "offline reputation", and that "number of published papers" is being used an an easily measurable proxy for this.

Cheers,

Matt

Dear Ben,

My impression is that this would be moving in the direction of presenting pure raw data, and leaving all interpretation and analysis to the reader. I think the goal of a study like this is to provide interpretation and analysis, not simply to provide underlying raw data. Obviously, one can question the interpretation and analysis (which is what you are doing); I am just trying to posit a reason (based on my own, limited, impressions of what is happening here) why Yla may not want to do what you are suggesting (perhaps at the expense of leaving the interpretation and analysis open to some criticism on this point).

Cheers,

Matt

A problem with Google Scholar is it does not always find the right person. For example, there are two people with the name John Harper that are both algebraic topologists. Some names are particularly bad for these kinds of issues, like Cohen.

I think people may want to compute their own h-indices because those are really used in promotional cases. Google scholar and "Publish or perish" are not easy to use, but one can in principle compute the h-index using these tools. As for the current survey, I agree that the total number of publications is not a reliable indicator of the off-line reputation. Moreover, there is no such thing as an objective "off-line reputation" because the reputation order is only partial and depends on who is ordering. Nevertheless there are many pairs of mathematicians $(A,B)$ for which almost everybody agrees that $A\gg B$ in any off-line reputation order. If you consider those pairs and the corresponding MO reputations you can easily conclude that there is very little correlation between MO and off-line reputations, as the survey claims. For example, M. Gromov and J.-P. Serre have combined MO reputation 0. This is not a big surprise: I think that in applied statistics, it is quite common that unreliable methods produce reliable results.