I think is worth drawing out the idea that even in contemporary mathematics there are notions which (so far) escape rigorous definition, but which nevertheless have substantial mathematical content, and allow people to make computations and draw conclusions that are otherwise out of reach.

I agree that this is a fascinating question, but I disagree that this is the question the OP is asking. (In fact, I don't know what the OP is asking! That's what I mean when I say that the OP is being imprecise.)

I think asking about definitions which are seen to be only unsatisfactory, approximative implementations of some idea is quite sensible. The fact that this occurs (and it surely occurs!) is an extremely interesting aspect of mathematical work; in particular, this is an aspect of maths that is completely missed by the popular view on the subject and on our work. But I think that the question in question should be made precise, if that is indeed what it is being asked.

A completely different question, «what are definitions that were unclear or unprecise?» in the sense of not completely defining a concept (I cannot think of any example of this right now) could also be asked... and right now I don't think it is at all clear which of the two is being asked!

E.g. despite Mariano's comment above, as far as I know there was no candidate definition of mixed motive prior to Voevodsky's work, and the definition of the abelian category of mixed motives (as opposed to its associated derived category) has still not been made. Nevertheless, there are many examples in the literature using the (not-yet-established) theory of mixed motives to draw significant conclusions.

In another direction, I remember when I was a student that there were many enumerative computations in algebraic geometry that could be made by computation in an appropriate quantum field theory, but which were not rigorously justified. Again, the situation has improved somewhat since then, but as far as I know there are still quantum field theoretic computations that cannot be made rigorously.

I also think that the hypothetical field F_1 is an example; despite the competing definitions that may have been offered, I don't think it is clear yet which (if any) of them is the correct.

There are other examples that I know of (e.g. the hypothetical Langlands group attached to a global field), and I'm sure there are examples that I don't know about as well.

In general, I think that it's an interesting state of affairs in mathematics when we have specific notions with strong predicative power which are not yet made rigorous, and it seems worthwhile to document some examples. If nothing else, it may implicitly offer suggestions to some readers as to where to direct their efforts.

I don't think we have to get into discussions about "what is a concept?", etc. The criterion is simply that there is a noun or phrase like F_1 that people talk about and have proposed rigorous definitions of, but there is not yet a unique definition (up to "cryptomorphism", to use a coinage of Rota).

The answers are mainly about ideas ("$q$-analogue", say), not with actual mathematical objects. The search for the correct category of motives does not mean that each of the candidates considered was not clearly defined, and so on.

The problem is not with the answers but with the question, which does not ask anything clearly defined...

I am reminded of Russell's tongue-in-cheek, quasi-Socratic remarks about it not being so obvious that one plus one equals two.

The original question blurs natural language with mathematics in a way that I think could lead to poor answers. It does not give examples (such as the one Kevin alludes to above) of what (s)he is after.

http://mathoverflow.net/questions/54932/how-were-moduli-spaces-defined-before-functors

What is a "notion"? When is it being "used"?

Yemon, at the risk of being facetious, if you honestly don't know the answer to these questions then I refer you to any English dictionary. (This is not to say that I disagree with your other objections.)

The serious point I'm trying to make is that words don't have to be defined to mathematical levels of precision if they're being used outside the technical context of mathematics.

1) In my view the answer to the question is arguably "everything" - the issue is what you mean by clearly defined. Structurally? Interpreted within a particular model?

2) What is a "notion"? When is it being "used"?

3) I think that it will be easy to give many (and frequent) poor answers to the question, with a somewhat subjective component. (If something isn't "clearly defined", could it be because you just haven't read the right literature?)

4) I do not think that the level at which David interprets the question is one which the questioner will greatly appreciate (but my opinion is coloured by previous questions (s)he has raised on MO).

A few examples: People at the n-category cafe seem to spend a lot of time discussing what the definition of an "infinity-category" should be. My understanding is that one of the reasons Lurie and Nadler-Ben Zvi's work is so exciting is that we seem to be getting close to agreeing on a definition. (Not my area, though, so I could be wrong.)

As I understand it, we don't have a rigorous definition of a "conformal field theory". (Again, not my area.)

Getting into combinatorics/combinatorial representation theory, I don't think there is a generally accepted definition of a "bijective proof" or of a "manifestly positive formula", which are things people in those fields spend a lot of time looking for.

Of course, there are also words like "trivial" and "elegant", but I think it is obvious that we don't want to define those!

I'm not sure this question will draw any great answers, but it might be interesting to see what other concepts people know of which are missing definitions.