With deeper penetration of distributed energy resources (DERs) in the distribution system, the power flow patterns are becoming more diverse, rendering over-current relays in distribution systems with potential malfunctioning. This paper proposes a data-driven approach to design operating strategies of relays in distribution systems with high penetration of distributed energy resources (DERs). A support vector machine-based classifier for the relay setting is introduced. Compared with conventional over-current relays, the proposed method could operate more accurately with smaller false alarm rate. It is also robust to altering system conditions such as load levels and fluctuating DERs. Numerical studies based on the IEEE 123-node test feeder illustrate the potential of this approach.