LibCompress
LibCompress is a compression and decompression library implemented entirely in WoW-friendly Lua. It supports the LZW and Huffman algorithms, and can automatically choose the most efficient algorithm for your data. One popular usage for this library is to send a compressed table to another player or add-on. Doing this requires additional encoding to remove the \000 characters from the data stream.
Take a look at the forum post for more info and a development discussion:
https://authors.curseforge.com/forums/world-of-warcraft/addon-chat/libraries/209315-libcompress
Usage:
Loading
The library is marked load on demand, so depend on it in your own addon or force load it using:
LoadAddOn("LibCompress")
Follow it with:
libc = LibStub:GetLibrary("LibCompress")
Compression
Load the library with:
libc = LibStub:GetLibrary("LibCompress")
Compress data (must be in string form):
compressed_data = libc:Compress(data)
This will try all compression algorithms and return the best compressed result. It is possible to specify a specific compression algorithm like this:
compressed_data = libc:CompressHuffman(data)
or
compressed_data = libc:CompressLZW(data)
Data will either be compressed with the Huffman compression algorithm or not at all. Data returned with a prefix byte identifying that the data is decompressed.
To decompress the data, simply use this:
decompressed_data = libc:Decompress(compressed_data)
Compress and Decompress can return an error and this is signaled by the first returned argument being nil and the second the error message. So checking for that would be appropriate.
Encoding
LibCompress also has the possibility to encode and decode data, preparing it for transmission over the addon channel or chat channel (or a custom encoding). Two forms of encoding is provided:
Prefix encoding
The first form is prefix-encoding. Basically, reserved characters are replaced with a prefix/escape character followed by the suffix character, i.e. reserved bytes are replaced by a double-byte combination. This is how it is done:
table, msg = libc:GetEncodeTable(reservedChars, escapeChars, mapChars)
reservedChars: The characters in this string will not appear in the encoded data. escapeChars: A string of characters used as escape-characters (don't supply more than needed). #escapeChars >= 1 mapChars: First characters in reservedChars maps to first characters in mapChars. (#mapChars <= #reservedChars)
If table is nil, then msg holds an error message. Otherwise the usage is simple:
encoded_message = table:Encode(message)
message = table:Decode(encoded_message)
Two predefined setups have been included:
GetAddonEncodeTable
: Sets up encoding for the addon channel (\000 is encoded)
GetChatEncodeTable
: Sets up encoding for the chat channel (many bytes encoded, see the function for details)
7-bit encoding
This encoding packs bits, not bytes. It puts 7 bits into every byte, enlarging the data by approx 14%. Values from 0 to 127 (both inclusive) are present in the encoded data and therefor has to be prefix-encoded as well. This encoding generates a bit of string trash and should be used with consideration.
Encode data like this:
encoded_data = libc:Encode7bit(data)
Decode data like this:
decoded_data = libc:Decode7bit(encoded_data)
Checksum/hash algorithms
LibCompress also provides 2 reasonable fast hash algorithms. They are converted from a C-implementation to lua and are quite fast. The hash value is either 16 bit or 32 bit.
Use like this (data1, data2, data... = string):
code = libc:fcs16init()
code = libc:fcs16update(code, data1)
code = libc:fcs16update(code, data2)
code = libc:fcs16update(code, data...)
code = libc:fcs16final(code)
data = string fcs16 provides a 16 bit checksum, fcs32 provides a 32 bit checksum.
line 676: code_high, code, code_len, _bitfield_high, _bitfield, _bitfield_len = getCode2(bitfield_high, bitfield, bitfield_len)
_bitfield_high, _bitfield, _bitfield_len are all globals, you may want to add them to your local declaration on line 657 to remove the taint its generating
Thanks. I must have missed them.
I'll fix this soon(TM).
I have committed the fix, but the packager doesn't seem to be working. Not sure what to do now. :-/
Edit. It was just lagged. Everything is fine now.
Hi there, galmok. I really, really need your help. I need to figure out how to decode a string encoded by GetAddonEncodeTable():Encode(str). Here's what's been achieved so far: https://www.reddit.com/r/wowaddons/comments/57y77y/are_there_any_opensourced_online_weakaura/ . Basically, the investigation didn't go any further than:
But could you explain that? decode_search seems to contain a string with a mix of unprintable characters and round/square brackets. decode_translate seems to be a table containing unprintable characters. I thought string.gsub() takes strings as parameters, yet there you give it a table. Also, what are those unprintable characters?
Is the only thing that GetAddonEncodeTable():Encode() does replacement of those unprintable characters? Because what I want to implement is a decoder (on my website) for WeakAuras strings. But they look more like Base64, yet not quite Base64. The noticeable difference is, encoded WA strings can contain brackets, and they seem to lack padding.
I don't know how you have to decode WeakAuras strings. But if they are stored using AceSerializer, compression and encoding, you have your work cut out for you if you want to reverse that in javascript. If LibCompress is used to store the data in the SavedVariables file, you will have to implement a Huffman decoder matching the bit pattern I used for building the Huffman compression. This is not exactly simply. On top of the, I guess WeakAuras also uses some for of serializer to be able to store tables and more. You will have to reimplement the deserializer in javascript as well.
Or, seeing as you have control over the website, maybe use the addons directly and have your webserver call Lua which loads the libraries and decodes the information using the LibCompress/AceSerializer themself?
Hello!
Just a quick note for those interested, you can use LibCompress, AceSerializer, and AceComms as an efficient way to send arbitrary data to other clients.
This code will prepare all the libraries needed:
Once you have the library objects set up (see above documentation, as well as the two relevant Ace pages), the following code takes an object and sends it:
On the other end, re-constructing the result applies the exact same procedure but in reverse. Since this could fail in various creative ways, this code is more involved:
Now, in this particular example, I only used one variable, and a string at that - so there was no real need to use the AceSerializer step. Thanks to LibCompress, sending a string in this manner is still better (length-wise, for any reasonably-long string) than sending it raw, however an advantage of AceSerializer is that any number of arbitrary data types can be thrown at it (tables, floats, etc) and it will be lossessly transmitted.
It shouldn't be too hard to modify this example to allow you to send many different data fields at once. The combination of these three libraries gives you a powerful method of transferring data.
Please note that I used CompressHuffman in this case, as it appears to be more length-efficient and faster for compressing (in my limited testing). LibCompress offers a feature that will try all compression algorithms available and use the smallest one: simply use the Compress function instead. This trades runtime (compressing each data set multiple times) for best-possible compression.
Even if it is just under consideration...
Also fixed the problem of HuffmanCompress not being able to handle the empty string as input.
Minor revision increased to 4.