Blueprint format/API?

Steve · Apr 20th 2017

Quote from Minotorious

...]Also a different question now:
I created two test cases one with a single ramp texture 21 and one with a single stair1 again texture 21 to test the block data structure.The values I get from the blueprint file are:

blocktype hex dec
ramp 26C1 9921
stair1 0F51 3921

seeing this result I understand that 21 refers to the texture, while the first two numbers refer to the position? If that is the case how does the game know if the block is a square block, a ramp or a stair1, a stair2, or a stair3 etc.?

Not where I can test this, but does the variation byte have any difference?

Quote from red51
Basically the first step would be to read the bytes of the blueprint file and decompress it (the game uses GZIP for compression). Once you have the decompressed byte buffer/array, you can start reading the single bytes. This is the structure of every blueprint (I hope I didn't miss anything - if you have any questions, don't hesitate to ask):

Bytes Info
1 blueprint version
2 blueprint size x
2 blueprint size y
2 blueprint size z
8 creation date (timestamp, amount of milliseconds passed since 1st of January 1970)
4 name length (String bytes)
xx blueprint name (byte length see previous 4 bytes)
4 author name length (String bytes)
xx author name (byte length see previous 4 bytes)
4 world name length (String bytes)
xx world name (byte length see previous 4 bytes)
4 block data length
xx block data (see below)

**** see "arrow" in modified code block ***

4 object data length
xx object data (see below)
4 construction data length
xx construction data (see below)
4 preview image data length
xx preview image data

[...]

Object data information

Objects (furniture, doors etc) are represented by a special object which holds information about the typeid (short), the variation (byte), position x y and z (float) and rotation x y z (short). The first 4 bytes in the encoded blueprint bytes hold information about the actual amount of objects in this blueprint. Java way to read the objects:
Java
//assumed the blueprint bytes are stored in a
//ByteBuffer called "buffer"


//get the total amount of bytes (4 bytes == int)
int olen = buffer.getInt();
byte[] objectdata = new byte[olen];


//fill the object byte array
buffer.get(objectdata);


//easiest way is to create a new ByteBuffer, although
//it would be faster to work with the blueprint buffer
//directly. However, we will take the easiest way here:
ByteBuffer objects = ByteBuffer.wrap(objectdata);


//get the amount of objects (4 bytes == int)
int amount = objects.getInt();


//create all objects. It's recommendable to create a
//separate class to hold the particular object information.
//Maybe create a new ArrayList or something like that to 
//store the objects
ArrayList<BlueprintObject> list = new ArrayList<>(amount);
for(int i = 0; i < amount; i++){
    BlueprintObject obj = new BlueprintObject();
    obj.typeid = objects.getShort();  //short (2 bytes)
    obj.variation = objects.get();  //1 byte
    obj.posx = objects.getFloat();  //float (4 bytes)
    obj.posy = objects.getFloat();
    obj.posz = objects.getFloat();
    obj.rotx = objects.getShort();  //short (2 bytes)
    obj.roty = objects.getShort();
    obj.rotz = objects.getShort();
    list.add(obj);
}


//////////////////////////////////////////


//our individual object class to hold blueprint objects:
public class BlueprintObject{
    public short typeid;




    public byte variation;                        <--------- This byte




    public float posx;
    public float posy;
    public float posz;
    public short rotx;
    public short roty;
    public short rotz;
}
Display More
Display More

Minotorious · Apr 20th 2017

@Steve I was talking about blocks (blocks are made in the blockbench and their names are block, ramp, stair1-3, half-block, cylinder, half cylinder, etc.) the code block you included above is for objects (objects are made in all other benches i.e. sawbench, workbench etc.). In game you can distinguish them when you use console commands to spawn them, blocks need the word "item" in front to be spawned while objects need the word "object".

e.g.:
item block 1 21
item ramp 1 21
object bed 1 0 <--this 0 is the variation e.g. this can be 0-10 i think for beds
object chair 1 0

red51 · Apr 20th 2017

Quote from Steve

I took the Pepsi challenge from one of your recent change logs on 400?x fly mode, using the native system JDK (Steam java is bogus on Linux, FYI.. I know the Ubuntu 2001 edition er 12.04 runtime and all...) and a few JVM tweaks for GC management to see if I could get screen tear or stutter

Oh, could you provide more information about that? Which JRE/JDK did you use exactly? Eventually we may replace the current JRE which is delivered with the Steam Linux version

Quote from Minotorious

wait wouldn't it be wiser to use the inbuilt method buffer.getShort() (i.e. blocks = buffer.getShort() ) instead of reinventing the wheel with bit shifting?

So true When writing my response I was originally working with a byte array instead of a buffer, so I wanted to avoid to wrap the array into a buffer But of course, if a buffer object is already available (like in this case), it d e f i n i t e l y makes sense to use the inbuilt buffer.getShort() method instead of messing around with bit shifting

Quote from Minotorious

I created two test cases one with a single ramp texture 21 and one with a single stair1 again texture 21 to test the block data structure.The values I get from the blueprint file are:

Now it's getting tricky: Currently the game uses a short buffer to store the block information, both the texture and the block shape are encoded into a single short value. Basically every shape has an individual "offset" (at least an offset of 300, to keep enough room for textures). In this case, 9900 is the block shape id, while 21 is the texture.
However, having both information encoded into a single value is crap. That's why we want to change it in the medium run (probably instead of using a single short array, we will use two byte arrays [one holding the texture id, the other one holding the shape id]).

Minotorious · Apr 20th 2017

thanks @red51 I now fully understood how blockdata works I will now run a few test cases to determine the unique identifiers for each type of block.
When you decide to change the blueprint file format I would appreciate if you could include the changes in the changelog or post them somewhere on this forum.

Edit: I forgot to also ask, is the typeid of each object the one I find in the definitions.db database in the game files?
i.e.:
1 rocksmall1
2 rocksmall2
3 tent
4 table
5 chair
6 shelf
7 bed
8 piano
9 workbench

10 torch
etc.

Steve · Apr 23rd 2017

Quote from red51

Oh, could you provide more information about that? Which JRE/JDK did you use exactly? Eventually we may replace the current JRE which is delivered with the Steam Linux version

See edit at bottom

So I've actually tested this game enough to ask for QC title. On Arch Linux ( the plain ole shoot yourself in the foot 5X then it runs for the next 10 years, Arch ) rolling release, 64bit,
currently on 4.10.10, but I have I think 2 kernel updates need to compile. I'm either the best Linux tester for you or the worst. Best b/c I know what info you're looking for.
Worst b/c on a rolling release I get updates within hours of them being available (Ubuntu might be 2 years.. not a dis, just facts, people So in a way it's like someone
futureproofing your code so long as it doesn't break before 50% of your Linux users catch up on libraries, etc. Ok, sorry JRE/JDK:

2017 JDK: (relevant timeline since I've played) Arch 64bit
===============================================

[2016-11-04] installed jdk (8u112-1) < I switch between these two. Probably 80-90% Oracle Java, but when testing
[2017-04-22] installed jdk8-openjdk-jetbrains (8.u112.b783.1-1) < JVM features, I would usually go test on both and they are 99% compatable I think.
+ The Oracle installs are binary install, while I compile the patched OpenJDK.
* Also on the laptop is Oracle early access 9 (snapshot), but obviously doesn't run RW.

I'll also probably be filing a but report tomorrow or Monday when I get a chance. My CPU usage went up with the latest release (why I was delayed reporting back here -- trying to figure out what it was before I blessed any given JDK... ) I've seemed to narrow it down to libfmod, but will need to install previous build to confirm. If there is something you can shoot back one line here, like "YEAH, we had a fmod update" that would make me feel better otherwise I'll gather some logs and such and let you grep your way through them.

2017 JDK: (relevant timeline since I've played) Arch 64bit [ host ] Windows XP 32bit, Windows Vista 64bit, Windows 7 64bit [ guests in Wine software]
======================================================================================================================
Before you ask -- it was to get some screenshots and test some setting for our server. I maintain a performance guide and it's tough getting people with time or give a s* to test config parameters like a proper group of Linux users would : ) JK Anyways, I ran all the above clients in Wine, so not quite the same as real Windows, but it was the RW Windows build, no
problem. Actually quite well.. Vista seems to be the best about 90% performance of native host. They were all tested using the Windows / Steam JRE/JDK. I wasn't about to try to install
java on a guest without a real VM in between. That just seems like a recipe for disaster : /

Probably more than you wanted, but that's it for now. I'll get you more details on possible regression from update, but I want to test the previous build first before I "cry wolf".***see edit

Thanks @red51

Edit 4/23

I did some more testing with the current version and found that of all things the FOV under View Adjustments made a huge difference going from 1.00 to 0.90. Obviously with less objects to draw, there was a noticeable increase in FPS, which didn't necessarily help the CPU usage (around 30%, 120-150 FPS). After adding vsync to bring it down (60 FPS), several benefits were immediately realized:

CPU usage dropped from 30% to 20% (previous release was around 15-18%, but will confirm)
CPU temp dropped from 71C to 57C (19% decrease) due to freq. decrease from turbo @3.4GHz to @2.8 GHz, which is about the base freq.

While I enjoy 1.0 FOV, I'll be using 0.90 for now.

Steve · May 2nd 2017

Quote from red51

Oh, could you provide more information about that? Which JRE/JDK did you use exactly? Eventually we may replace the current JRE which is delivered with the Steam Linux version

Thank you! I noticed today how soon "eventually" means in Red-time

Bytes	Info
1	blueprint version
2	blueprint size x
2	blueprint size y
2	blueprint size z
8	creation date (timestamp, amount of milliseconds passed since 1st of January 1970)
4	name length (String bytes)
xx	blueprint name (byte length see previous 4 bytes)
4	author name length (String bytes)
xx	author name (byte length see previous 4 bytes)
4	world name length (String bytes)
xx	world name (byte length see previous 4 bytes)
4	block data length
xx	block data (see below) ** see "arrow" in modified code block *
4	object data length
xx	object data (see below)
4	construction data length
xx	construction data (see below)
4	preview image data length
xx	preview image data

Participate now!