Overfitting occurs when a model performs well on training data but fails to generalize to new data. Proper evaluation uses a hold-out test set, with metrics like Mean Absolute Error for regression and confusion matrices for classification to balance precision and recall, ensuring real-world model effectiveness.