Αποστολή με SMS: Sample Efficient Multiagent Learning in the Presence of Markovian Agents