Yep can confirm it gets worse as you play and explore and place items. The server just doesn't seem to handle it or something. Host sees the stuff and can connect pretty easily, the rest of us place invisible objects, and whenever you interact ( ie chop a tree down --- light a fire ) then it's invisible.
This is the biggest issue I've come across so far, because ultimately you can't get very far into the game, our first server we ditched because we found a huge cave and we legit can't load back into the game, it's impossible.