LMeve - reduce your memory footprint

hello fellow industrialists.


Just a quick post for those of you who are running a VM for LMeve, with a very limited amount of memory.

As I was toying with a VM running with only 128mb of RAM, I noticed that my MySQL DB, which was empty at that moment, was already using more than 128mb of RAM+swap. I decided to take a look on how to reduce it memory footprint.
That DB will be used only by a few handmade scripts, with a very low bandwidth usage and access, very similar to the LMeve usage for a one man corporation, or a couple of industrialist in a small corp.

After a few tests, I came up with these numbers, which are, in my case, enough. Bear in mind that I'm not a DBA, so, I've a limited understanding in DB bits and bytes. But here's what I found (feel free to correct me if I'm wrong)

• innodb_buffer_pool_size is one of the main parameter to deal with MySQL inner buffers pools in memory. Reducing its size will reduce its RAM usage.
• join_buffer_size and sort_buffer_size are used while standard operations are taking place (select, read, write, etc). Since I've little use of them, I see no point in having a lot of memory devoted to that.
• read_rnd_buffer_size is used in conjunction with sort_buffer_size, so I set them at the same size. 

With those values, I've seen no impact on my low memory VM. I may change them if performance are bad when the DB will have more rows. (if it impacts your LMeve, restore the .orig, and/or, tune those values)

The last parameter is the one that was using an insane amount of memory for my empty and low b/w DB. I needn't it to be fast, I will write/read stuff a few times per hour, as for LMeve. So, I disable those performance schema.
After those modifications, my empty MySQL which was using more than 128mb in memory+swap combined has disappeared from the swap usage.


# cp /etc/my.cnf /etc/my.cnf.orig
# vi /etc/my.cnf
  edit:
innodb_buffer_pool_size = 16M
join_buffer_size = 4M
sort_buffer_size = 128k
read_rnd_buffer_size =128k

# disable Performance Schema (free up memory) - in the [mysqld] section.
performance_schema = off

  Save and exit.

# systemctl restart mysqld
# for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | grep -i mysql
 mysqld_safe 0 kB
mysqld 0 kB
#

An other "optimization", is to reduce buffer to active connections, since I know that I won't have more than two active connections at a given time:

# mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.6.28 MySQL Community Server (GPL)

Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show variables like "max_connections";
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| max_connections | 151   |
+-----------------+-------+
1 row in set (0.00 sec)

mysql> set global max_connections=10;
Query OK, 0 rows affected (0.00 sec)

mysql> commit;
Query OK, 0 rows affected (0.00 sec)

mysql> show variables like "max_connections";
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| max_connections | 10    |
+-----------------+-------+
1 row in set (0.01 sec)

mysql> exit
Bye
#

coming soon (tm)

Hi there,

I forgot to mention that with the release in mid december 2015 of Operation Frostline, and my research for profitable items, I noticed that I was unable to find some TypeIDs on some items. That's when I realised that I didn't update my LMeve nor TypeID list since July 2015, to reflect to various tiericide and new items added in-game since past summer.
I did an upgrade of LMeve to 0.1.54, and the CCP's DB to the Operation FrostLine release.
Sadly, I did it without taking too much notes, thus, I have no How-To Upgrade LMeve.

For the Citadel release, or whatever name CPP will find for it, I will post a complete walkthrough to update LMeve and CCP DB.

In the meantime, I could give you two hints about installing LMeve:

• use a generic name for the CCP's DB. I used "eve_aeg100_dbo" for mine, since it was the release of Aegis 1.00, I should have used a more generic one :-) 
• I finally find why YAML imports weren't working properly at my inital LMeve install. On CentOS/Redhat, the PHP configuration has some default values that are too small for the YAML import to succeed. In the /etc/php.ini, change the memory_limit from 128M to 1024M.

Oh, and I noticed that Lukas (LMeve author) has found this blog and post a link to it. Thanks for the great tool you have provided us :-)

Status update for early 2016

Hi there,

Long time I haven't post here, I was too busy in RL to pay much attention to the blog.

On the Eve side, I moved to a low sec system a couple of months back, moving all my assets to
that system, planing a lot of safe warp point, insta warp-off and insta-dock. Over the course
of those few months, I lost only a couple of empty T1 industrials on the warp gates. Mostly due
to my careless jump before checking if it was safe on the other side.

I decided on Monday to move everything back to Hi-Sec, since I almost lost my newly bought Freighter.
As I warped my scout to the other side, I checked d-scan and local for a couple of seconds, then
jumped my freighter. As I pressed Jump, I noticed a t2 frigate in warp to my gate then landing on it !
As my scout was still holding his cloak, and that pilot surely noticed the gate activating, I kept the cloak on my freighter, and lure the frigate to a warp to the sun. Sure thing, he did followed me, but instead, I was warping to a safe in direction of the sun, so, he landed alone overthere, while I was warping out of the system via that very same gate :-)
In the meantime, as we were warping to the sun, I decloaked the freighter and warped it slooooooowly to my insta-dock at the station. That was somewhat a close call. I was already pondering about moving out before this episode, I was now convinced that I had to move out.
My low-sec system is a secondary staging system for a low-sec alliance, which is at war since a few months with some big null-sec alliances. If you follow the news in Eve Online, you already heard about that.

So, well, that episode told me that I have to move everything back to HiSec. At least, now, I have a freighter (and still have :-) ), and moving my stuff won't be as painful as it was in last October. My last production batch was due to finish this Friday morning, so, by the end of this weekend, it should be done.

With that part almost done, I'm focusing in optimizing my operations in HiSec. My main issue is that my base of operation is pretty far from my trade hub, which mean that I'm back with long travel time to ship my production to the tradehub. Since I've scaled up, it's roughly 3 to 4 millions of m3 per batch, or 3 to 4 full freighter haul.

To help me with my planification, I built a tentacular excel spreadsheet, feeding autonomously from various websites and from my LMeve DB. It keep track on my production costs, per system, per BPO according to their research level, my stocks in minerals, salvage parts, my operating costs, an historic of my transactions (with graphs).
I'm working on that spreadsheet since august or july 2015, and I keep adding new stuff almost weekly. I know that most of that job has been done already by people who have a way better understanding of the game mechanics than me, but this is my way to have a better grasp at Eve, to fully understand its complexity. I don't want to use something perfect built by someone else, I want to have a faulty tool, built by me, which I will improve as my knowledge improve.

I already revamped most of the sheets at least twice, to add taxes, deductions, automate feeds and so one. I had to learn VBA for excell (I'm an old school ASM and C guy), PHP to deal with LMeve, re-learn most of my SQL knowledge from 20 years back, and how to deal with excel. I would say that I spend more time toying around with excel than playing.
My in-game interactions are limited to buying minerals, negotiating contracts to move from point
A to point B or moving items myself, setting my build and sell orders, that's it.

Boring ? not at all in my opinion.
Some people like shooting at little red crosses (or those new icons for NPCs), others at hunting people in low/null secs, at exploring, at space-trucking, at trading and so one. Eve Online is a sandbox, you could do whatever you want if you like that kind of activity.
That's why Eve Online is such a powerfull game.

Be creative, fix yourself an objective, and go for it. (Oh, and please, get blown up doing it, that's a game after all)

That's it for now.
I have some further planifications to do for the weekend :-)

Fly safe.