Hi.
I think you've summarized the situation very clearly, but sometimes attraction can be so powerful that we try to talk ourselves out of what we already know.
I'm sure you genuinely admire your teacher--and the forbidden aspect of having an affair with him may even be adding to the excitement--but it may help to think about the reality of the situation.
A relationship with your teacher could ruin or seriously damage his career. Teachers who have affairs with students are usually fired or, if the student is under the age of sexual consent (generally 16-18, depending on the state), may be charged with statutory rape.
If he's fired because of this, he may feel resentful toward you and may discontinue seeing you because the relationship is too damaging to his life. If he is put in jail, you will no longer have access to him anyway.
Relationship-wise, there is probably a big difference between your life experience and your teacher's, even if he is on the youngish side---say, mid-twenties. He has gone through college and has probably had many experiences you've yet to experience. That may pose an imbalance in the relationship.
Such a relationship could also make things really difficult with your parents, if it is exposed.
There are just gobs of things that can go wrong. Try limiting your contact to class time and see if that helps. Going by his office every day will only add to your suffering.
Take care.