I don't think closed questions should be hidden by default. If we wanted that, we would just delete them. Part of the point of having closed questions hang around is to provide a record of the kinds of questions that aren't appropriate for the site. The point of closing a question is that you can no longer answer it or vote on it, so it doesn't generate reputation and it mostly doesn't get pushed to the top of the pile (comments don't push the question to the front page). Editing the question still pushes it to the top of the front page, but that is the desired behavior. Perhaps the edit cleans the question up to the point that it will be reopened (something that usually takes five votes from high reputation members of the community).

I haven't seen a situation where closed questions keep getting pushed to the top of the pile, so I'm not too worried about this. If you do come across a situation where it's getting out of hand, flag the post for moderator attention, with a note asking a moderator to lock the post. This prevents any non-moderator activity on the post. Unfortunately, it also prevents people from leaving comments, which is why we don't lock all closed questions.