Unions for instructions

[jdpatdiscord]

When looking at the Luau source back when it released around Nov 3rd, I was surprised to see that Luau did not use unions for storing instructions, when I had successfully used unions in my own private projects to represent them.

My findings were that despite different compilers having limitations to how well a union could be packed with something as complicated as a Luau instruction, it ended up being "good enough" where it was able to encode every possible format into a 32-bit union, except for one minor inconvenience.

Unions would be a notable improvement in code cleanliness, and I'm happy to share my findings on union packing Luau instructions specifically if it is an interest. (clang/msvc/gcc)

[jdpatdiscord]

@zeux I don't think you were notified. Should I move this to an issue for better visibility?

[zeux]

I've read this but I guess it wasn't clear that this requested a response :) I don't think unions would result in a noticeable improvement in readability and since some instructions in Luau are multi-word there's an odd conflict between either actually using a larger union to represent them or still using implied layout via several union reads.

This shouldn't stop you on elaborating on your findings of course, but it's not clear why we should change our existing approach.

[jdpatdiscord]

Thankfully it can be made complex enough to make the actual programming simple.

#pragma pack(push, 1)
	/* i A {B C {Bx sBx}} */
	union LuauInst
	{
		struct
		{
			uint8_t i;
			// Note: MSVC doesn't pack structs like CLANG/GCC: Do not wrap Ax with A/B/C, it will generate a 5 byte struct
#ifndef _MSC_VER
			union
			{
				signed sAx : 24;
				unsigned Ax : 24;
				struct
				{
#endif
					uint8_t A;
					union {
						struct {
							uint8_t B;
							uint8_t C;
						};
						uint16_t Bx;
						int16_t sBx;
					};
#ifndef _MSC_VER
				};
			};
#endif
		};

		uint32_t Dword;
	};
#pragma pack(pop)

I've been using this exact struct in my code for nearly a year, and it has caused no problems cross platform. It eliminates the need for macros or bitshifting, and (hopefully) the compiler can know what to do with it better. The only workaround on MSVC is you do instr.Dword << 8 to access what would be instr.Ax in this example