You might not fear a simple branch or leaf, but such things can turned into a hazard to those swimming within your pool.
The heat lost to your air is obvious, but any water that evaporates has also used energy when being warmed or chemically fixed.