Text this: Sample Efficient Multiagent Learning in the Presence of Markovian Agents