Yeah. Going along with a big ass, sinister looking robot with anger issues while it builds an army of robots seems like a smart thing. Especially since during scuffles with the Avengers Ultron just fucks around while Wanda and Pietro are the ones actually fucking them up. Which means they don't even need him. Them working with him feels to me like a script note "three sides are harder to follow, how about the bad guys working together?"