Dev Diary #205 Automated Testing

[Nikki Severin 1148]

Hey there, Ylanders!
 

We are now fully focusing on bringing you adventures set below the waves and we're already seeing some pretty cool stuff that we'll be showing you in the upcoming weeks. This is how it always is quite early in the development of an update - things are being designed, a detailed document made, prototypes created... so not quite something that can be presented to the player base (especially since some of these things often end up being removed from the update).

But what we're working on are not only things that you players will experience first hand. Today we would like to mention a tool/system that our programmers have been working on for some months now - Automated Testing. Ylands is already an incredibly complex project and adding any new feature, regardless of how carefully done, many times ends with something breaking someplace else in the code. Regardless of the hundreds of hours that QA puts in testing before every update is released, it is not humanly possible to test all the situations which may arise. And this is where Automated Testing comes in. Even though the system still needs a few months to be fully ready, it's already possible to create various tests and run them regularly to see if something didn't break.

Imagine this very simple test - a few objects are placed in the level and the character is told to pick them up and then we test if they are all in the character's inventory. This could be one of hundreds of tests that run automatically and report only when the test outcome isn't as expected. It could be anything - a problem with reaching the objects due to pathfinding, objects disappearing, problems with picking up objects, or with the inventory. If a problem is found, it is reported and QA can check it (for example see a recorded session to find what went wrong). This will hugely (probably starting somewhere before 1.10 is released) improve the game stability and the number of issues present.
 

(screenshot by YoHasLEGO)

Automated tests or not, the most important thing we have apart from our internal QA is ... you. You have reported hundreds of bugs and the absolute majority of these have been fixed. Your feedback matters and the game is now so much better because of all the time you have spent talking to us. For that, we are extremely grateful.

So this is it for today and we'll see you next week when we're gonna start talking about some juicy Exploration stuff.

Stay safe and classy!

[Mello1223 591]

i want this joypad in game ? 

[zarwil 393]

For people who don't know much about these things, it is impossible to build software in a sustainable manner without explicitly testing every function and module extensively throughout development, however that's probably not what this post is referring to specifically.

I think what this actually means is that BI is adopting a continuous integration pipeline with automated unit tests, as is industry standard (at least in general software development). Basically, individual developers can't run tests that cover the entire code-base every time they make a tiny change, so they are mostly constrained to testing smaller subsets of the code during day-to-day work. After that you hope and pray that the tests are comprehensive enough to cover unexpected interactions with the rest of the code-base. With automated tests in a continuous integration pipeline, you have a dedicated server specifically tasked with running tests for the entire code-base (or perhaps subsets of it). At the end of the day, for instance, any developer can deploy their modified code to the CI-server and be notified with a detailed report whenever the tests are complete. If you don't have a system like this in place, integrating different parts of the software can be an absolute nightmare. 

[Miguel Preguisa 443]

47 minutes ago, zarwil said:

For people who don't know much about these things, it is impossible to build software in a sustainable manner without explicitly testing every function and module extensively throughout development, however that's probably not what this post is referring to specifically.

I think what this actually means is that BI is adopting a continuous integration pipeline with automated unit tests, as is industry standard (at least in general software development). Basically, individual developers can't run tests that cover the entire code-base every time they make a tiny change, so they are mostly constrained to testing smaller subsets of the code during day-to-day work. After that you hope and pray that the tests are comprehensive enough to cover unexpected interactions with the rest of the code-base. With automated tests in a continuous integration pipeline, you have a dedicated server specifically tasked with running tests for the entire code-base (or perhaps subsets of it). At the end of the day, for instance, any developer can deploy their modified code to the CI-server and be notified with a detailed report whenever the tests are complete. If you don't have a system like this in place, integrating different parts of the software can be an absolute nightmare. 

Ha ha, this comment made me once again feel like a psychology freshman reading through my first academic papers. The feeling of "I thought I do understand English. All of these words are seemingly English, but yet..." ? It takes me a second read-through for me to switch gears to this kind of text. 

[zarwil 393]

24 minutes ago, Miguel Preguisa said:

Ha ha, this comment made me once again feel like a psychology freshman reading through my first academic papers. The feeling of "I thought I do understand English. All of these words are seemingly English, but yet..." ? It takes me a second read-through for me to switch gears to this kind of text. 

It's difficult to be concise without using some industry jargon hahah ?