Well, he wouldn't have to go there every day. Just once, like the others using the Burrow. He could go somewhere else other days and Floo from there (or he could probably Floo from Buckingham if no one could overhear him).

But Mrs Longbottom thinks it's too dangerous so it's okay.