Optimizing memory usage in LUA

Couple of days ago I received a message on curse.com about the memory usage of UberInventory. The general statement was that UberInventory uses a lot of memory. We were aware of this issue, but we had never given it much attention. But after this message we could ignore it no longer.

We have been reading several articles on the internet on how to optimize memory usage, performance and garbage collection using LUA. All those documents basically state that using locals as much as possible improves performance when accessing data structures and when storing a lot of data most of the times it is better to create larger tables instead a lot of smaller tables. This some times leads to a little more unreadable data structures but it keeps garbage generation to a minimum and less memory is needed to store the data.

Since UberInventory contains a lot of static data (pricing and recipe data) our first focus was to see if and how we could optmize those data structures.

The following results have been generated using LUA 5.0.1 and we have used the gcinfo() function to produce the numbers.

Orignal version of UberInventory_prices.lua

	Dynimac Memory	Garbage threshold
Before	1531	3062
After	4189	6124
Delta	2658	3062

UBI_Prices = {
 [25] = {nil,7},
 [35] = {nil,9},
 [36] = {nil,7},
};

Modified version of UberInventory_prices.lua

	Dynimac Memory	Garbage threshold
Before	1060	2121
After	2500	2121
Delta	1440	0

UBI_Prices_Buy = {
 [38] = 1,
 [39] = 5,
 [40] = 5,
};
UBI_Prices_Sell = {
 [25] = 7,
 [35] = 9,
 [36] = 7,
};

As you can see with the modified version there is big drop in the total memory that is needed to store the data and the structure does not create any additional garbage.

Changes made to UberInventory_prices.lua

• Split data into two tables, one for buy prices and one for sell prices

Original version of UberInventory_recipes.lua

	Dynimac Memory	Garbage threshold
Before	961	1921
After	3160	3842
Delta	2199	1921

UBI_Creatures = {
 [66] = { name='Tharynn Bouden', area='Elwynn Forest' },
 [340] = { name='Kendor Kabonka', area='Stormwind City' },
 [450] = { name='Defias Renegade Mage', area='Westfall' },
};
UBI_Quests = {
 [22] = { name='Goretusk Liver Pie', area='Westfall' },
 [38] = { name='Westfall Stew', area='Westfall' },
 [90] = { name='Seasoned Wolf Kabobs', area='Duskwood' },
};
UBI_RecipeDrops = {
 [4213] = { {8716,1}, {12396,1} },
 [4408] = { {7800,1} },
};

Modified version of UberInventory_recipes.lua

	Dynimac Memory	Garbage threshold
Before	636	1272
After	1230	1272
Delta	594	0

UBI_Creatures = {
 [66] = 'Tharynn Bouden|Elwynn Forest',
 [340] = 'Kendor Kabonka|Stormwind City',
 [450] = 'Defias Renegade Mage|Westfall',
 [590] = 'Defias Looter|Westfall',
};
UBI_Quests = {
 [22] = 'Goretusk Liver Pie|Westfall',
 [38] = 'Westfall Stew|Westfall',
 [90] = 'Seasoned Wolf Kabobs|Duskwood',
};
UBI_RecipeDrops = {
 [4213] = { '8716:1', '12396:1' },
 [4408] = { '7800:1' },
};

Here you see the same results, huge drop and no garbage. These changes combined resulted into a decrease in memory of between 2 and 3 MB just for the static data.

Changes made to UberInventory_recipes.lua

• For creatures and quests combine the name and area into one text field instead of using a table structure
• For drop info combine mob id and drop rate into one text field instead of using a table structure

All the changes described in this post are implemented for UberInventory 1.6.

The next step will be digging into the data structures of the dynamic data stored within UberInventory.lua (WTF\Account\…\SavedVariables) and optimize those for memory usage and garbage.

Replacing HDD in an iRiver iH120

Years ago I was an avid user of an iRiver iH120, but since I started using iPod’s (yes, I have multiple) it was becoming a dust bunny. When my girlfriend started to follow some weekend courses, I gave the iH120 to her so she could record the lectures so if she had missed something during the course or just wanted to review a particular lesson she could just play the lesson to review it.

It as always worked until the iH120 was accidentally drop from about 1 meter onto a hard floor. The HDD did not survive that drop. So I bought her a Creative ZEN 4GB. It is a very nice MP3 player and has the ability to be used as a recorder. However it turned out to be not such a great recorder after all. It records the sound very well, but it has the tendency to pick all the sound in the vacinity making the recordings very hard to understand.

So I decided to see if I could restore the good old iH120 to working order, assuming it was just the harddrive that was broken and not any other parts of the player. So I did a bit of searching to see what kind of harddrive was needed for this type of player. It truned out that a Thosiba MK3006GAL harddrive of 30GB could be used as a replacement drive. So after this small repair it would have 10GB more space than before the drop.

I ended up buying on from gebruiktelaptop.nl where I could get one for € 19.95, which was quite surprisingly cheaper than what people were asking for the drive on eBay (lowest € 24, highest € 52).

This week the drive finally has arrived and time to get out the tools, well just one tool. You only need one Torx screwdriver (TX5) to take apart the iH120 and replace the HDD.

The replacement process
First a total of eight screws need to be removed, four from the bottom and four from the top of the iH120, after which you can remove the plastic top and bottom covers.

Next you carefully open the unit making sure that you do not damage or brake the power cable that connects the two halfs of the iH120 together.

Now the old drive can be replaced with the brand new HDD and the iH120 can be assembled again.

The whole process takes about 5 to 10 minutes to complete, and no technical knowledge is required.

Booting the iH120
Now that all the ‘hard’ work was completed it was time to boot the iH120. At first all seem to be fine the iRiver logo appears and after a while the main screen appears. However when using the browse function some strange info appeared on the screen.

So I shutdown the iH120, opened it again, made sure all the cables were properly connected. All appeared to be fine so I closed the unit again. Then I decided to hook the iH120 up to my Mac and access the internal HDD of the iH120 using a USB cable. The iH120 showed up without any problems and I could copy files back and forth between the iH120 and the Mac. Conclusion HDD is working fine. As last resort I started Disk Utility to do some further testing of the HDD and as soon as I had a look at the HDD info within Disk Utility the problem was clear. It turned out the HDD was formatted using NTFS instead of FAT32. So I reformatted it as FAT32, disconnected the iH120 from the Mac.

This time when I booted the iH120 all was in working order and I am again a happy man :)

Keeping my digital life in-sync (3)

This is the third installment of my quest to keep my digital life in-sync. In the second part about my quest I blogged about the use of Dropbox. In this post I am going a little deeper into some of the advantages I have come across on the internet.

I am using the application 1Password by Agile Web Solutions to keep track of all my username/password combinations used for the many websites I make use of. Up until a few days ago I used the Mac OS X Keychain synchronization of Mobile Me to synchronize the data between my two Macs. On the Agile Web Solutions Wiki I came across a page describing how you can use your Dropbox folder to store and synch the Agile keychain between your machines.

I have been using it for a couple of days now and so far it works very well, and feels a lot faster when data needs to be synchronized.

As soon as I heard of the existence of Things by Cultured code I installed on my Mac and am using it as my main application to track all the things I need to do or have done by a certain time. This worked very well but it became cumbersome when I bought a second Mac (a Mac Book Pro) and started traveling around. There was a way of manually copying certain files to a USB stick and constantly copy these files between the two Macs. This goes well for a certain amount of time, but eventually I forgot to do the copy trick and was stuck with two difference lists of things.

Method #1: The article Sync Your Things Database via Dropbox desrcibes how you can move the Things library to a different location and then create a symbolic link on the original location for the datafiles that links back to the new location. Read this article carefully, if you don’t feel comfortable using the Terminal use the next method.

Method #2: I was very pleased when I came across the following wiki page describing how Dropbox can be used store it at a location that can be synchronized between several machines. This method is easier then method #1

The only things you really need to be aware of is that you only have one copy of Things running on any machine and make sure synchronizing has been performed between the machines before using Things on the other machine.

Method #1 as described for synching the Things database can be used for other Mac applications as well (for those applications that not let you choose where their information is being stored, like with 1Password), so I am making a list of applications for which I want to keep the data synched between machines.

I am liking Dropbox more and more everyday.

UberInventory 1.6

We are currently working on UberInventory 1.6. At the time of this writing we have not yet set any release date.

Following is a list of tasks we are working on;

• Fix issue that occurs on fresh installs (or joining a new guild) of UberInventory when visiting the guild bank

• Redesigning the settings module
• Adding more options to customize the behavior of UberInventory
• Improving overall performance (specifically determine usability of items)
• Removing obsolete code