Although GPS has become a standard component of smartphones, providing accurate navigation during the last portion of a trip remains an important but unsolved problem. Despite extensive research on localization, the limited resolution of a map imposes restrictions on the navigation engine in both indoor and outdoor environments. To bridge the gap between the end position obtained from legacy navigation services and the real destination, we propose FOLLOWME, a "last-mile" navigation system to enable plugand-play navigation in indoor and semi-outdoor environments. FOLLOWME exploits the ubiquitous, stable geomagnetic field and natural walking patterns to navigate the users to the same destination taken by an earlier traveler. Unlike existing localization and navigation systems, FOLLOWME is infrastructure-free, energy efficient and cost-saving. We implemented FOLLOWME on smartphones, and evaluated it in a four-story campus building with a testing area of 2000m2. Our experimental results with 5 users show that 95% of spatial errors during navigation were 2m or less with at least 50% energy savings over a benchmark system.