After a while I would think Felicity would need to talk to him and ask him what the hell was going on. She's pretty smart and the need to know would be a key driver, along with making sure that the working relationship wasn't damaged, at which point things with Oliver could progress... with the right questions and answers...

And from there some smut would be great. ;-)