Intelligent consumer energy management systems will become important elements at the delivery points of the smart grid inside homes, buildings, and industrial plants. The end users will be able to better monitor and manage their energy consumption, while utilities will gain more flexible mechanisms for management of peak demands that will extend beyond demand response initiatives as they are implemented today. With a broader use of distributed generation many buildings and campuses will become microgrids interconnecting multiple generation, storage, and consumption devices of one or several end users. We discuss how energy management and control for such facilities can be viewed as a large-scale optimization problem. Specific supply-side and demand-side aspects include on-site renewable generation, storage technologies, electric cars, dynamic pricing, and load management. Technical challenges related to the optimization formulation are noted - in general, mixed-integer, nonlinear, constrained optimization is needed. We also describe an implementation of optimization-based energy management solution for a hospital in the Netherlands, providing economic details and an analysis of the savings achieved.