It is not a sin to recognize the beauty of another. But when we look, and then look again and again, or imagine lustful things, then we have crossed the line. Each person knows in his heart when he if falling short of purity.
In the early Church, the story has been told of a few bishops gathered outside a cathedral. On the sidewalk walking by was a prostitute, dressed very much like one. Several bishops put their heads down to avoid seeing her, while another bishop stared straight at her. As she passed, a bishop with his head down looked to the other bishop and said, "Brother bishop, you can't look. It's a prosti-" And then he saw tears streaming down the cheeks of the bishop, who said "How sad, that such beauty should be sold to the lust of men." The prostitute saw how he looked at her, and do you know what she did? She converted, and she's now a saint in the Church: St. Pelagia. The purity of that one man's glance was enough to elevate her back to God.
Looking at a woman with the eyes of Christ is true purity. If we are not ready to do this, and we are too easily tempted to lust, then we better look elsewhere as God continues to purify our hearts.
Is it possible that a girl's feelings could be hurt if we look away? Yes. But maybe she needs that wakeup call so that she'll know what she's worth.