At the time I wasn't with you
By that time I didn't care
A very personal line. Taylor seems to be speaking from a past personal experience with this girl rather than just a story with two unknown characters.
Taylor could have had feelings for this girl when they weren’t seeing each other and by the time they were together and he had finally got what he wanted he didn’t care for her anymore.
To help improve the meaning of these lyrics, visit “Bowery” by Local Natives Lyrics and leave a comment on the lyrics box