Custom game saves and packets

User avatar
Alundaio
S.T.A.L.K.E.R.
Posts: 1371
Joined: 26 May 2012, 22:26

Custom game saves and packets

Postby Alundaio » 29 Nov 2013, 16:09

I had a theory on how to combat save game corruption and also allow modders to save vast amounts of custom information without overflowing or corrupting the game save, like what can happen with my dynamic spawn system. Turns out my theory is indeed very possible.


Here is the class I wrote that emulates a packet (Note: I have zero previous experience with this):

Code: Select all

class "sstpk"
function sstpk:__init()
   self.header_count = 0
   self.size = 0
   self.data = ""

   self.w_e = 0
   self.r_e = 0

   self.w_i = 0
   self.r_i = 0
end

function sstpk:w_header(id,size)
   local sz = size or self:w_elapsed()

   self:w_seek(0)
   self:w_u32(id)
   self:w_u32(sz)

   self.header_count = self.header_count + 1
end

function sstpk:r_header()
   self:r_seek(0)
   self.header_count = self.header_count - 1
   return self:r_u32(),self:r_u32()
end

function sstpk:clear()
   self.header_count = 0
   self.size = 0
   self.data = ""

   self.w_e = 0
   self.r_e = 0

   self.w_i = 0
   self.r_i = 0
end

function sstpk:flush(path)
   local f,e,d = io.open(path,"wb")
   if not (f) then
      alun_utils.printf("%s,%s,%s",f,e,d)
      return
   end

   self:w_seek(0)
   self:w_u32(self.header_count)

   f:write(self.data)
   f:close()
end

function sstpk:parse(path)
   local f,e,d = io.open(path,"rb")
   if not (f) then
      alun_utils.printf("%s,%s,%s",f,e,d)
      return
   end
   self:clear()
   self.data = f:read("*all")
   self.size = f:seek("end")/2
   f:close()
end

function sstpk:w_tell()
   return self.size
end

function sstpk:r_tell()
   return self.size
end

function sstpk:w_begin()
   self.w_e = 0
   self.w_i= 0
   if (self.size == 0) then
      return
   end
   self:w_seek(self.header_count*8)
end

function sstpk:r_begin()
   self.r_e = 0
   self.r_i = 0
   if (self.size == 0) then
      return
   end
   self:r_seek(self.header_count*8)
end

function sstpk:w_elapsed()
   return self.w_e
end

function sstpk:r_elapsed()
   return self.r_e
end

function sstpk:w_seek(n)
   self.w_i= n*2
   self.w_e = 0
   if (self.w_i> self.size*2) then
      self.w_i= self.size*2
   elseif (self.w_i< 0) then
      self.w_i= 0
   end
end

function sstpk:r_seek(n)
   self.r_i = n*2
   self.r_e = 0
   if (self.r_i > self.size*2) then
      self.r_i = self.size*2
   elseif (self.r_i < 0) then
      self.r_i = 0
   end
end

function sstpk:write_data(str)
   local sz = string.len(str)

   local a,b = string.sub(self.data, 0, self.w_i), string.sub(self.data,self.w_i+1)
   self.data = string.sub(self.data, 0, self.w_i) .. str .. string.sub(self.data,self.w_i+1)

   self.size = self.size + sz/2
   self.w_i = self.w_i + sz
   self.w_e = self.w_e + sz/2

   alun_utils.printf("write=".. a .."["..str.."]".. b)
end

function sstpk:read_data(typ,bytes)
   if (bytes == nil or self.data == nil or self.data == "") then
      return nil
   end

   local a,b,c = string.sub(self.data,0,self.r_i),string.sub(self.data,self.r_i+1,self.r_i+bytes*2),string.sub(self.data,self.r_i+bytes*2+1)
   alun_utils.printf("read= "..a.."["..b.."]"..c)

   local ret = string.sub(self.data,self.r_i+1,self.r_i+bytes*2)
   if (ret == "" or ret == nil or ret == "nil") then
      return nil
   end

   self.data = string.sub(self.data,0,self.r_i)..string.sub(self.data,self.r_i+bytes*2+1)

   if (typ == 0) then
      ret = tonumber(ret,16)
   elseif (typ == 1) then
      ret = alun_utils.hex2string(ret)
   elseif (typ == 2) then
      ret = tonumber(alun_utils.hex2float(ret))
   end

   --self.r_i = self.r_i + bytes*2
   --print(self.r_i)
   self.size = self.size - bytes
   self.r_e = self.r_e + bytes*2

   return ret
end


function sstpk:w_bool(val)
   val = val and 1 or 0
   self:write_data(string.format('%02X',val))
end

function sstpk:r_bool()
   local ret = self:read_data(0,1)
   return ret == 1 and true or ret == 0 and false
end

function sstpk:w_u8(val)
   val = val and val >= 0 and val <= 255 and val or 255
   self:write_data(string.format('%02X',val))
end

function sstpk:r_u8()
   return self:read_data(0,1) or 255
end

function sstpk:w_u16(val)
   val = val and val >= 0 and val <= 65535 and val or 65535
   self:write_data(string.format('%04X',val))
end

function sstpk:r_u16()
   return self:read_data(0,2) or 65535
end

function sstpk:w_u32(val)
   val = val and val >= 0 and val <= 4294967295 and val or 4294967295
   self:write_data(string.format('%08X',val))
end

function sstpk:r_u32()
   return self:read_data(0,4) or 4294967295
end

function sstpk:w_u64(val)
   val = val and val >= 0 and val <= 18446744073709551615 and val or 18446744073709551615
   self:write_data(string.format('%016X',val))
end

function sstpk:r_u64()
   return self:read_data(0,8) or 18446744073709551615
end

function sstpk:w_float(val)
   val = val or 0
   self:write_data(alun_utils.float2hex(val))
end

function sstpk:r_float()
   return self:read_data(2,5) or 0
end

function sstpk:w_stringZ(val)
   val = val or "nil"
   local l = string.len(val)
   if (l > 65535) then
      val = string.sub(val,0,65535)
      self:w_u16(65535)
      l = 65535
   else
      self:w_u16(l)
   end
   self:write_data(alun_utils.string2hex(val))
end

function sstpk:r_stringZ()
   local l = self:r_u16()
   return self:read_data(1,l) or "nil"
end



For example if we used it to do this:

Code: Select all

sstpk:w_stringZ("actor")
sstpk:w_u8(255)


It would look like this for each step, the hex values within the brackets are what is being written:

Code: Select all

[0005]               -- length of the string being written
0005[6F626A5F31]         -- hexcode of the string being written
00056F626A5F31[FF]      -- hexcode for the unsigned 8-bit integer



Then if the packet was done, we would write the header using a unique id, which can be the object id:

Code: Select all

sstpk:w_header(0)


It writes two 32-bit unsigned integers, one for the id and the other for the elapsed size of data we just wrote:

Code: Select all

[00000000]00056F626A5F31FF
00000001[00000008]00056F626A5F31FF


Write more info:

Code: Select all

sstpk:w_stringZ("npc")
sstpk:w_u8(1)

000000010000000800056F626A5F31FF[0003]
000000010000000800056F626A5F31FF0003[6E7063]
000000010000000800056F626A5F31FF00036E7063[01]



Need another header:

Code: Select all

sstpk:w_header(1)

[00000001]000000000000000800056F626A5F31FF0003
00000001[00000006]000000000000000800056F626A5F31FF00036E7063
0000000100000006000000000000000800056F626A5F31FF00036E706301



Then we write how many headers this packet has, which is only two at the moment:

Code: Select all

sstpk:w_seek(0)                  -- seek the beginning of the packet
sstpk:w_u32(sstpk.header_count)      -- write header count as a 32-bit uint


Looks like this:

Code: Select all

[00000002]0000000100000006000000000000000800056F626A5F31FF00036E706301


So now the packet should look like this:

Code: Select all

000000020000000100000006000000000000000800056F626A5F31FF00036E706301


This can be done for each object that needs information to save. The headers will keep track of who and how much. When the data is being read it will make it easier to sort which bytes belong to what. First you would need to read the header, then use r_begin() to jump to the starting point of the real information, then read in that much according to the size written in the header.



The packet can be written to with the flush command:

Code: Select all

sstpk:flush("some_directory\my_save.dat")



Then loaded up like this:

Code: Select all

sstpk:parse("some_directory\my_save.dat")





Now my implementation is a bit more complicated then the examples above because I want to be able to have separate packets for each object that subscribes for one and then merge them together into a single net packet that is saved. It looks something like this right now and is currently located in se_actor:STATE_Write function:

Code: Select all

   local se_obj
   local net_sstpk = stpk_utils.net_sstpk()
   net_sstpk:clear()
   for i=65534,0,-1 do
      se_obj = sim:object(i)
      if (se_obj and se_obj.sstpk_STATE_Write) then
         alun_utils.printf("sstpk for id=%s",i)
         if not (se_obj.sstpk) then
            se_obj.sstpk = stpk_utils.sstpk()
         end

         se_obj.sstpk:w_seek(0)
         se_obj:sstpk_STATE_Write(se_obj.sstpk)

         if (se_obj.sstpk.size > 0) then
            net_sstpk:w_seek(0)
            net_sstpk:w_header(i,se_obj.sstpk.size)

            net_sstpk:w_begin()
            net_sstpk:write_data(se_obj.sstpk.data)

            se_obj.sstpk = nil
         end
      end
   end

   if (net_sstpk.header_count > 0) then
      net_sstpk:w_seek(0)
      net_sstpk:w_u32(net_sstpk.header_count)
   end


Basically what it does is loops through all possible object ids and check if the object exists. IF it does exist and it has a sstpk_STATE_Write function, it will create a new packet, trigger that function and pass it to the object where it can be manipulated. Then the packet is merged into the main net packet where it can be later saved.

The bonus is, when you later load that data in and cannot find the object, you can toss out that data! It won't affect or overflow the other information as long as it was written properly.


I do this for saving, which is in ui_save_dlg.script:

Code: Select all

   if nil~= fileName then
      local console = get_console()
      console:execute("save " .. fileName)
      if (stpk_utils) then
         alun_utils.printf("save file: %s",alun_utils.fspath("$game_saves$")..fileName..".dat")
         stpk_utils.net_sstpk():flush(alun_utils.fspath("$game_saves$")..fileName..".dat")
         axr_main.last_save = fileName
      end
   end
end


As you can see, it will save a .dat file with the same name as the save!


I don't have any experience working with packets, game saves or anything like that. All this is just based on how I think it should work, which it does so far. This theory is panning out very well and can be a huge step in modding. I'll be able to have my entire dynamic spawn system independent of the actor's packet and maybe port existing saving over to have more control over the values being saved/loaded to prevent corruption.
"I have a dream that one day this community will rise up and live out the true meaning of its creed: "We hold these truths to be self-evident; that all mods are created equal."

User avatar
Alundaio
S.T.A.L.K.E.R.
Posts: 1371
Joined: 26 May 2012, 22:26

Re: Custom game saves and packets

Postby Alundaio » 30 Nov 2013, 16:43

I had to over come an issue with loading a new game from a fresh launch. Only scripts that are called upon by the ui_main_menu.script are loaded in the main menu screen and then all scripts are re-loaded when a game is started. There can be no communication between these two phases meaning it's impossible to know what game is loaded without externally storing this information. Luckily, I already have written .ini reading/writing functions in lua. I store the name of the last save into a config file when the game is loaded, then read it on game start.

I have writing/reading and saving/loading of my packet data working. I have been unable to find a good way to get this to work they way I initially wanted, which was to allow any object to subscribe for SSTPK manipulation. I might just have to take the easy route and only use this process in the bind_stalker:save() and bind_stalker.load(). It's hurting my brain trying to think of a way.


EDIT:

;) I got it working, exactly the way I wanted. Will save data for anything that subscribes to the event. Now time to implement it for my dynamic spawn.
"I have a dream that one day this community will rise up and live out the true meaning of its creed: "We hold these truths to be self-evident; that all mods are created equal."

User avatar
Alundaio
S.T.A.L.K.E.R.
Posts: 1371
Joined: 26 May 2012, 22:26

Re: Custom game saves and packets

Postby Alundaio » 01 Dec 2013, 16:49

Update:


Dynamic spawn seems to be working well with it. I did run into a problem with slow saving if you save tons of stuff (Like 228Kb). String manipulation is very slow in lua so I need to find a way to make this faster. There is string concatenation every time I add data to the "packet". lua tends to allocate new memory for each string being concatenated which adds a lot of work for garbage collection which can make the game freeze.

This shit is pretty cool. I can save whatever I want.
"I have a dream that one day this community will rise up and live out the true meaning of its creed: "We hold these truths to be self-evident; that all mods are created equal."

User avatar
david.m.e
Trespasser
Posts: 84
Joined: 24 Oct 2012, 20:34

Re: Custom game saves and packets

Postby david.m.e » 05 Dec 2013, 06:55

It sounds like it's gonna make a wonderful base for building on, I'm really looking forward to using it. Once you have something ready if you want someone to play around with it looking for bugs then I'll do some testing for you.

r_populik
Scavenger
Posts: 47
Joined: 07 Aug 2012, 03:22

Re: Custom game saves and packets

Postby r_populik » 05 Dec 2013, 08:06

Yeah, I could do some testing too. I'm going to test my weapon mod, with AI tweaks it would be a lot more exciting.

Swartz
Adventurer
Posts: 200
Joined: 27 May 2012, 08:37

Re: Custom game saves and packets

Postby Swartz » 07 Dec 2013, 14:24

Make that three people, I can do testing as well.

User avatar
Alundaio
S.T.A.L.K.E.R.
Posts: 1371
Joined: 26 May 2012, 22:26

Re: Custom game saves and packets

Postby Alundaio » 08 Dec 2013, 14:50

Thanks for testing, I appreciate it. With a game as random as this with so many things I've added it's very hard to test it all. Flooding in and breaking shit helps me fix it faster. :P
"I have a dream that one day this community will rise up and live out the true meaning of its creed: "We hold these truths to be self-evident; that all mods are created equal."


Return to “General Discussion”

Who is online

Users browsing this forum: No registered users and 1 guest