RESOLVED [YLD-11416] Servers grow in lag, crash, freeze, require wipe

[RedEagle_P1. 905]

In all cases on all server types, when playing the stock explore game type servers grow in lag as more and more people play. The more people do things, the more it lags and it needs to be wiped to fix.

[RedEagle_P1. 905]

@bojo2736 Can you send them an overloaded map that needed to be wiped and was unstable on servers? 

[bojo2736 1016]

Sure. How many do you want?

[RedEagle_P1. 905]

5 hours ago, bojo2736 said:

Sure. How many do you want?

The more the better  

 

[bojo2736 1016]

This is the most recent map on our NA server.  Created as a standard explore map with some simple scripting added.  Uploaded around 8/5/19   It went belly up on Thursday or Friday 8/23/19  It was 31 mb and is corrupted I think. I can't open it and it won't run on the Nitrado server.

Pretty typical for MP maps that approach 30 ish mb.  Which they can do in just a couple of days. Depends on how active things are.

Sorry, I didn't get logs. I was busy making a new map which will last two weeks. 

082419.zip

[bojo2736 1016]

This is one before that. In the end it was abandoned by my players as being totally unplayable.  This one is the largest to date. But I think it had been abandoned for a week.  This was at a time when we were waiting for the ship fix.  I think that's why it was an extra week past the normal 3 week maximum I have ever seen.

I can give you more. I keep the files. And they are literally ALL this way.   If it wasn't for the fact that I still have a few die-hard players, I would have abandoned this game months ago.  In terms of my personal play, I already have.  Y'all should check out Outlaws of the Old West. 

NA_07_09.19__VANILLA-BACKUP-V02_190711024748.zip

[Indomitus 388]

I'm not one of the devs so I don't know for sure, but my (non-game) developer instincts are crying out "data indexing problem."  It's reminding me of issues I used to have with older database formats, with the file size growing quickly and performance taking a nose dive.

It does kind of make sense to me, if players are digging up resources (spawning new entities) and exploring to new chunks that were either not fully defined or not defined at all when the map was created.  The server would probably append that data to the end of the game file, or the end of certain blocks of data.  Over time, the indices can get muddled, and lookup operations in the code get slower and slower.  It can even result in corrupted data.  And the problem does happen on local single-player games as well, it just takes a whole lot longer to get that bad.

IF that's the case, then fixing it for the long term will be a pain.  I don't envy them on that job.  However, maybe something could be added to server options to take a map offline and rebuild the indices, on demand or on a schedule, that could ease the pain until a full fix is complete.

And just for the sake of saying it again:  I don't know that is the problem, really.  This is just reminding me of issues I've faced with older business software I've built.

[Adam Snellgrove 1451]

So this is the Big One. I am going to really try and find out, what's going on with this, but so far I have information, that stuff is being done. We decided to dedicate our next Dev Diary to this (and to bugs and what we plan to do about them in general), so I'll try and find out more about this till then and report back my findings.

[bojo2736 1016]

What I have done to reduce file size and extend server life is to take the file into editor and remove any extraneous items.  Hope my players don't need seaweed and rocks.  I have tried optimizing the file, but what happens is that players can no longer build on their houses, can't paint, it just becomes locked.  If there could be a construction switch, like the ships have, but maybe on the power barrier.  Players could weld and unweld their own spaces. This would also, hopefully allow them to make things unpickable in game as well.

I have not been able to open the last two files in editor. It just nopes out on me. 

[RedEagle_P1. 905]

5 hours ago, Adam Snellgrove said:

So this is the Big One. I am going to really try and find out, what's going on with this, but so far I have information, that stuff is being done. We decided to dedicate our next Dev Diary to this (and to bugs and what we plan to do about them in general), so I'll try and find out more about this till then and report back my findings.

Good idea! This is not the only part of it though, its also desync. Also that logic-breaking-over-time one I reported started happening in just 15 min of play so it can't be related to this. 

[Adam Snellgrove 1451]

1 hour ago, RedEagle_P1. said:

Also that logic-breaking-over-time one I reported started happening in just 15 min of play so it can't be related to this.

I'll make a separate issue for this and ask around.

[bojo2736 1016]

I am struggling to continue caring about this game.

I had issues on my standard explore map.  So I pulled it into editor.  I have been doing this for over a year that I have been admin leader of P1 PVE games.  When I loaded up the game, it had removed my previous character.  Replaced me with a new character.  This means I lost access to my power barrier.  As well as all the admin tools I had on the map.

I can't get back in to fix problems in editor explore.  And now I can't get in to fix things in standard explore.  This game is too broken to have it run without being able to get in and fix problems.  I have been doing these maps for over a year.  I have over 6000 hours in the game.  I have purchased multiple copies for myself and friends.  I used to love this game.  I'm just about ready to delete it from my computer.  I could give the files, but what would the point be?
 

Edited September 4, 2019 by bojo2736

[MyPa553ng3r 72]

We have been chasing our tails for a while.  Constantly going back and forth between types of games, Vanilla or Scripted Editor Explore.   We start a vanilla map, thinking our scripts are the culprit, only to have the map bloat the first 24 hours, and we are quickly reminded why we were using editor made maps with scripts that we use to try and fix the issues in the vanilla game.  Then we use a scripted map, with the same end, just different means, and go back to a vanilla map, to start the cycle over.  I'm starting to find comfort in the ritual.

 

[bojo2736 1016]

I'm thinking maybe it's just a social experiment. To see how much bullspit we will tolerate.

[kimbuck 746]

Today   was like another  charlie chaplin   movie .. screen  freezes and re starts

I got my ship  built and  tried   sailing... whenever it was near land it was uncontrollable .. the screen   kept  freezing and when release i was either heading  to shore  or out to sea...  then   ship went into a series  of slide show  images  showing  either  no ship,  just the wake ,  bits of the ship ..or   a part assembled  ship ..or   just me with my  bottom skimming the waves.

Then  Ylands  did a  perfect  impression of a  dead  goldfish .... belly up!

Was locked on the log out  screen for 5  minutes  as the wheel of  misfortune  slowly  turned.

I managed to log  back in to find myself on the ship    full  sail towards the  end of the world ... did a 180  turn  and  eventually  mad it back to my island... but as soon as i could see the shore,   screen  once again went into freezing  fits.

Managed to  coax it towards  dock on half  sail  then  hit reverse .. screen froze .. got the   sailing back   noise  and when  scree became  active i was halfway up the beach .. but going backwards... fortunately  into  deep water.

Given up for a while  ... just toooooooooooo   frustrating.

Logs and one screenshot attached

 

 

output_log_clean.txt

output_log.txt

[Adam Snellgrove 1451]

I'll look into it kimbuck ?

5 hours ago, bojo2736 said:

I could give the files, but what would the point be?

If you do, it'd help us get rid of the problem. This is still an Early Access game and some stuff isn't finished or isn't working, but we're working on it and especially after 0.14 (which will be very soon), we'll be directing all our manpower on bugs and server issues. We are at the finishing straight and if we manage to catch all things buggy and fix them, I believe you'll find it in your heart to love Ylands once more ?

If you could make a topic of this issue and tell us, what's wrong and maybe attach the output_logs, we'll get right on it.

[kimbuck 746]

Guess what?

I went to log into the latest  game on NA1 ..it  98%vanilla...  and  ....grrrrrrrrrrrrrrrrrrrrrrrr

The same issues  as with the previous  map that  went  bottom up.

The game loaded to recalling buildings 100% and i got that  thump and the  humm from the PB..and then a short message   "building oceans" .......

and then the wheel of misfortune  ground almost to a halt ....     two segments ..freeze     then  3  segments ...freeze.....   one segment...freeze       rinse and repeat .. then  "time out"

After the  third  attempt at starting form the  very beginning, i tossed my  hands in the   air  and  gave up.

No logs or  screenshots ..... as i couldn't get in far enough to  grab any

And only   1 day in the game ...  (sigh)

off to paint my nails ...  and look for  lint in my navel....

i say no  more?

 

[Adam Snellgrove 1451]

14 minutes ago, kimbuck said:

Guess what?

I went to log into the latest  game on NA1 ..it  98%vanilla...  and  ....grrrrrrrrrrrrrrrrrrrrrrrr

The same issues  as with the previous  map that  went  bottom up.

The game loaded to recalling buildings 100% and i got that  thump and the  humm from the PB..and then a short message   "building oceans" .......

and then the wheel of misfortune  ground almost to a halt ....     two segments ..freeze     then  3  segments ...freeze.....   one segment...freeze       rinse and repeat .. then  "time out"

After the  third  attempt at starting form the  very beginning, i tossed my  hands in the   air  and  gave up.

No logs or  screenshots ..... as i couldn't get in far enough to  grab any

And only   1 day in the game ...  (sigh)

off to paint my nails ...  and look for  lint in my navel....

i say no  more?

 

This sounds serious. Please also make a topic for this. I'll ask around what could be the problem and try to get a fix ASAP.

[kimbuck 746]

There is another server    P1EU  run  by BBcakes ...on a bi server   i  have no issues  there.

This latest thing  happens as soon as i  move a  created ship. And  on  relogging it  hangs after recalling buildings.... loading   ocaeans ..

 

[Adam Snellgrove 1451]

I've moved the issue here: https://ylands.com/community/topic/25386-yld-18598-loading-trouble/?page=0#comment-78590

 

[Adam Snellgrove 1451]

So. Aleš says all we need are output_logs from a server, that grows unplayable if it is rented from us. If it is a server you're running and it starts to be unplayable, send us the save file. And in both instances DO NOT RESTART THE SERVER until we give the get-go. It might be quite an easy thing to fix apparently...but also maybe not ? ? But lets try this and the programmers will have a look at it as soon as they recieve these files (well during working hours obviously ?)

[Adam Snellgrove 1451]

@bojo2736 I actually have some news on this! So the team found out (and this will sound bizarre), that the nr. 1 reason the servers grew slower and slower was, that animals fell through the ground sometimes and tried to find a way back up, but they couldn't. And the more animals fell through, the more animals were trying to find their way up again, which put considerable strain on the servers. 

Now we are fixing it and it should be part of the next bigger update.

Hopefully, this will increase server survivability considerably, but it'll be a case of looking for the next big thing, that slows down servers when this gets fixed.