Optimizing memory usage in LUA

Couple of days ago I received a message on curse.com about the memory usage of UberInventory. The general statement was that UberInventory uses a lot of memory. We were aware of this issue, but we had never given it much attention. But after this message we could ignore it no longer.

We have been reading several articles on the internet on how to optimize memory usage, performance and garbage collection using LUA. All those documents basically state that using locals as much as possible improves performance when accessing data structures and when storing a lot of data most of the times it is better to create larger tables instead a lot of smaller tables. This some times leads to a little more unreadable data structures but it keeps garbage generation to a minimum and less memory is needed to store the data.

Since UberInventory contains a lot of static data (pricing and recipe data) our first focus was to see if and how we could optmize those data structures.

The following results have been generated using LUA 5.0.1 and we have used the gcinfo() function to produce the numbers.

Orignal version of UberInventory_prices.lua

	Dynimac Memory	Garbage threshold
Before	1531	3062
After	4189	6124
Delta	2658	3062

UBI_Prices = {
 [25] = {nil,7},
 [35] = {nil,9},
 [36] = {nil,7},
};

Modified version of UberInventory_prices.lua

	Dynimac Memory	Garbage threshold
Before	1060	2121
After	2500	2121
Delta	1440	0

UBI_Prices_Buy = {
 [38] = 1,
 [39] = 5,
 [40] = 5,
};
UBI_Prices_Sell = {
 [25] = 7,
 [35] = 9,
 [36] = 7,
};

As you can see with the modified version there is big drop in the total memory that is needed to store the data and the structure does not create any additional garbage.

Changes made to UberInventory_prices.lua

• Split data into two tables, one for buy prices and one for sell prices

Original version of UberInventory_recipes.lua

	Dynimac Memory	Garbage threshold
Before	961	1921
After	3160	3842
Delta	2199	1921

UBI_Creatures = {
 [66] = { name='Tharynn Bouden', area='Elwynn Forest' },
 [340] = { name='Kendor Kabonka', area='Stormwind City' },
 [450] = { name='Defias Renegade Mage', area='Westfall' },
};
UBI_Quests = {
 [22] = { name='Goretusk Liver Pie', area='Westfall' },
 [38] = { name='Westfall Stew', area='Westfall' },
 [90] = { name='Seasoned Wolf Kabobs', area='Duskwood' },
};
UBI_RecipeDrops = {
 [4213] = { {8716,1}, {12396,1} },
 [4408] = { {7800,1} },
};

Modified version of UberInventory_recipes.lua

	Dynimac Memory	Garbage threshold
Before	636	1272
After	1230	1272
Delta	594	0

UBI_Creatures = {
 [66] = 'Tharynn Bouden|Elwynn Forest',
 [340] = 'Kendor Kabonka|Stormwind City',
 [450] = 'Defias Renegade Mage|Westfall',
 [590] = 'Defias Looter|Westfall',
};
UBI_Quests = {
 [22] = 'Goretusk Liver Pie|Westfall',
 [38] = 'Westfall Stew|Westfall',
 [90] = 'Seasoned Wolf Kabobs|Duskwood',
};
UBI_RecipeDrops = {
 [4213] = { '8716:1', '12396:1' },
 [4408] = { '7800:1' },
};

Here you see the same results, huge drop and no garbage. These changes combined resulted into a decrease in memory of between 2 and 3 MB just for the static data.

Changes made to UberInventory_recipes.lua

• For creatures and quests combine the name and area into one text field instead of using a table structure
• For drop info combine mob id and drop rate into one text field instead of using a table structure

All the changes described in this post are implemented for UberInventory 1.6.

The next step will be digging into the data structures of the dynamic data stored within UberInventory.lua (WTF\Account\…\SavedVariables) and optimize those for memory usage and garbage.

Replacing HDD in an iRiver iH120

Years ago I was an avid user of an iRiver iH120, but since I started using iPod’s (yes, I have multiple) it was becoming a dust bunny. When my girlfriend started to follow some weekend courses, I gave the iH120 to her so she could record the lectures so if she had missed something during the course or just wanted to review a particular lesson she could just play the lesson to review it.

It as always worked until the iH120 was accidentally drop from about 1 meter onto a hard floor. The HDD did not survive that drop. So I bought her a Creative ZEN 4GB. It is a very nice MP3 player and has the ability to be used as a recorder. However it turned out to be not such a great recorder after all. It records the sound very well, but it has the tendency to pick all the sound in the vacinity making the recordings very hard to understand.

So I decided to see if I could restore the good old iH120 to working order, assuming it was just the harddrive that was broken and not any other parts of the player. So I did a bit of searching to see what kind of harddrive was needed for this type of player. It truned out that a Thosiba MK3006GAL harddrive of 30GB could be used as a replacement drive. So after this small repair it would have 10GB more space than before the drop.

I ended up buying on from gebruiktelaptop.nl where I could get one for € 19.95, which was quite surprisingly cheaper than what people were asking for the drive on eBay (lowest € 24, highest € 52).

This week the drive finally has arrived and time to get out the tools, well just one tool. You only need one Torx screwdriver (TX5) to take apart the iH120 and replace the HDD.

The replacement process
First a total of eight screws need to be removed, four from the bottom and four from the top of the iH120, after which you can remove the plastic top and bottom covers.

Next you carefully open the unit making sure that you do not damage or brake the power cable that connects the two halfs of the iH120 together.

Now the old drive can be replaced with the brand new HDD and the iH120 can be assembled again.

The whole process takes about 5 to 10 minutes to complete, and no technical knowledge is required.

Booting the iH120
Now that all the ‘hard’ work was completed it was time to boot the iH120. At first all seem to be fine the iRiver logo appears and after a while the main screen appears. However when using the browse function some strange info appeared on the screen.

So I shutdown the iH120, opened it again, made sure all the cables were properly connected. All appeared to be fine so I closed the unit again. Then I decided to hook the iH120 up to my Mac and access the internal HDD of the iH120 using a USB cable. The iH120 showed up without any problems and I could copy files back and forth between the iH120 and the Mac. Conclusion HDD is working fine. As last resort I started Disk Utility to do some further testing of the HDD and as soon as I had a look at the HDD info within Disk Utility the problem was clear. It turned out the HDD was formatted using NTFS instead of FAT32. So I reformatted it as FAT32, disconnected the iH120 from the Mac.

This time when I booted the iH120 all was in working order and I am again a happy man :)