My impression is that this would be moving in the direction of presenting pure raw data, and leaving all interpretation and analysis to the reader. I think the goal of a study like this is to provide interpretation and analysis, not simply to provide underlying raw data. Obviously, one can question the interpretation and analysis (which is what you are doing); I am just trying to posit a reason (based on my own, limited, impressions of what is happening here) why Yla may not want to do what you are suggesting (perhaps at the expense of leaving the interpretation and analysis open to some criticism on this point).

Cheers,

Matt

I think that Yla already explained that the desired variable is "offline reputation", and that "number of published papers" is being used an an easily measurable proxy for this.

Cheers,

Matt

Thanks for making this comment. In particular, I support the sentiment of your third paragraph.

Regards,

Matthew

Yes, number of papers is measuring something: the number of papers. You have collected this data and analyzed its correlation with ratings of MO submissions: sounds good. This is what has actually been done. I don't see the added value in equating "number of papers" with "offline reputation", as neither a theoretical framework for the latter is being presented nor an argument for the validity of the former as a measurement for the latter. You could explain that you chose to measure and analyze the number of papers out of a feeling that is capturing something about the status of the participants in the mathematical community, but that's about it.

You mention the problem that offline reputation is a concept which is being used in more general contexts where "number of papers" doesn't make sense. I understand and respect that concern, but I don't see how that just picking something and calling it offline reputation contributes to the solution.

I repeat my request for pointers to literature on the theoretical concept of "offline reputation", which Ms. Tausczik said is a standard term in her field. Without this, I don't see how it is possible to discuss the validity of counting papers as a measuring tool.

responding to your earlier question

Also, do you think number of published math papers is a poorer measure of offline reputation than MO Points is a measure of online reputation?

could I first ask for clarification of what you mean by "online reputation" and "offline reputation" in the first place? I'm not really sure how I would try to define these separately. Perhaps we could try to do this as the respect held for someone on account of the actions performed in either an "offline" or "online" context. Thus while I'm deeply impressed by, say, T, as a mathematician, in my regard he has accrued some "offline" reputation when I've seen him give excellent public lectures, and some "online" reputation when he's posted great articles on his blog. But if I read one of his papers via the arxiv, and like it, does that contribute to his "online" or "offline" reputation? And if a trusted colleague tells me via email "Wow, T's latest result is fabulous", which is that?

I suspect that with a definition of online and offline reputations along these lines, I would argue that MO reputation is an extremely good proxy for "online reputation", but merely because the main other available source of it is writing blog posts, which relatively very few mathematicians do.

One of the main critics of operationalism in social science argues that "the original goal was to eliminate the subjective mentalistic concepts that had dominated earlier psychological theory and to replace them with a more operationally meaningful account of human behavior. But, as in economics, the supporters ultimately ended up "turning operationalism inside out" (Green 2001, 49). "Instead of replacing 'metaphysical' terms such as 'desire' and 'purpose'" they "used it to legitimize them by giving them operational definitions." Thus in psychology, as in economics, the initial, quite radical operationalist ideas eventually came to serve as little more than a "reassurance fetish" (Koch 1992, 275) for mainstream methodological practice."

and indeed it seems to me that that's what's going on here. Rather than replacing the fuzzy concept of "offline reputation" that one might initially think was of interest with the "operationally defined" concept of "number of published papers", you're conflating the two.

Number of papers clearly is measuring something, because it predicts some amount of the ratings of submissions on MathOverflow.

Sorry to be pugnacious about this, but why then don't you call number of published papers "online reputation", rather than "offline reputation"? It seems you have a better case for this misnomer than the one you're using!

Of course this is being a devil's advocate, it would be silly to actually call number of published papers "online reputation". It's just that it seems to me that calling it "offline reputation" is equally inappropriate as it would be to call it all sorts of other things. Surely there are some criteria for deciding when "operationalization" has gone too far? (I admit of course, that I am totally ignorant of psychologists' practice here.)

"I hope we can all agree that there is a clear difference in the amount of offline reputation that an undergraduate student has, and that a full professor has, and that the number of papers published will show that."

No, not yet. First we need to agree on what offline reputation means. If it actually means "number of published papers", yes, it will show that. But as I said above, you seem to be relying on some more vague idea of what the term means. One can find undergraduates with more published papers than full professors. For instance, Sophie Morel is a full professor at Harvard University and has, according to MathSciNet, one published paper. So no, your implicit equation is really not so clear.

"In fact, I think it's likely that we could run the same analysis with offline reputation being a binary measure of whether or not the user had any published papers, and get similar results."

Then perhaps you should try it and see.

"As far as "irresponsibility", it is common practice within psychology to try to measure a (ill defined and difficult to measure) feature of the real world by making an operational definition, and then naming the measure by the feature of the real world. Others can then argue about how good that operational definition is, but nobody thinks or claims that an operational definition completely captures the real world feature, or that it should be interpreted as the real world feature."

The word "irresponsibility" is a little bit strong here. But I don't entirely agree with what you say: that which is ill-defined deserves more efforts at attempted definitions and descriptions, not less. Similarly with things which are difficult to measure. In other words, it seems professional to explicitly address and argue for your equation of "offline reputation" and number of published papers. Moreover, I think the portion of your paper when you set off a red boxed number one and write "Offline reputation: Number of published math papers" is definitely a claim that your operational definition captures the real world feature.

I do think it is facile to present "measurements" without carefully addressing the extent to which the quantity that you're recording is actually a measurement of your phenomenon. To say "others can then argue" about it is kind of a copout, because that's not how things work in real life or even in academic life: academics have been told that "they can argue about" the actual meaning of student evaluation scores, impact factors and NRC rankings. Well, anyone can argue about anything but that doesn't change the fact that these statistics are now being used in making the most important decisions: hiring, promotion, tenure, funding, and so forth.

"Perhaps it only explains 10% of the "Platonic ideal offline reputation", but within my field it is standard to call it "offline reputation" even in this case."

Again, you seem to be presuming (perhaps facetiously, but that's kind of the point) that there is some Platonic ideal of offline reputation. What does this term actually mean to you? Moreover, in what academic field does the "offline reputation" have a meaning so standard that it appears in papers without definition or question? [I would be interested to see references.] Could this term really be more than a few years old? You're truly okay with reporting a measurement that might explain only 10% (or less?) of what it's desired to measure? Shouldn't you at least try to do better?

Here is a suggestion: why don't you measure offline reputation by a survey? But be sure to tell the surveyants what the term means!

There are plenty of undergraduates that publish papers (e.g. through REUs), but they probably don't have the "offline reputation" (or experience, or ability to answer MO questions) that a fourth-year graduate student at a top school that (for whatever reason) never published would.

I'd say the number of published papers is such a wildly imperfect measure of offline reputation that it is irresponsible to use the phrase "offline reputation" to refer to this number.

@Angelo- I guess you're just suggesting that the name of that variable be changed to "Number of published papers" ?