05 March 2008

Doing Splits on a Log

So I turned on my computer today and saw something along the lines of "starting mysql.....failed (not enough space on partition /dev/sda2)". That means / was really close to full. I nearly DoS'd myself. Awesome. The first place to look when / is suddenly full is /var. It's called "var" because of its variable size. Specifically, look in /var/log/. You can use the Disk Usage Analyzer (aka Baobab) in GNOME to figure out where the problem is, assuming you can reach GNOME. If your disk is so full you can't get into a GUI or if you use KDE (which I know nothing about), du -csh * will tell you the total size of each file/directory in the current directory including everything contained inside them. It's an extremely useful command. When I looked there, I found about 3GB of logs in /var/log/mythtv/. I starting playing with MythTV a few days ago because I'm setting it up for a school project.

While logrotate was kind enough to split the logs into 300-400MB chunks, that's still huge. At least, it's still too huge for vim to open the log in any reasonable amount of time. Enter the "split" command. A quick look at the manpage and I decided to be lazy about it and just ran split mythtvbackend.log. As per the defaults, I ended up with 1000-line files named xaa, xab, xac,...xzy, xzz, and then it complained it had run out of 2-letter combinations. Whatever, I didn't care. I only needed to be able to see inside one of them. Check the manpage if you want to change how many characters it makes available at the end for naming thousands of files it creates.

A quick peak inside the first of those showed that MythTV was looking for its default password instead of the one I set for it. Well, that's easy enough to fix in the Mythbuntu Control Center. I guess I shouldn't have ignored it when MythTV kept telling me it couldn't reach the database...oh well, live and learn.

My system administration teacher says "didn't I mention making a separate /var partition?" Yes, yes, but I didn't expect to be running any log-producing services on a laptop like I would be on a server.


Yuriy said...

Re: "or if you use KDE"
On Kubuntu/KDE 3.5 (I don't think this is included in default KDE/konqueror, but I'm pretty sure it's available by default on Kubuntu) you can open up a folder in konqueror, and go to file size view to get a graphical representation of what's taking up how much space.

Anonymous said...

Er, why use an editor to view a log file?

Try one of these:

less logfile
more logfile
tail -f logfile

Justin said...

du -axc / | sort -n | tail -n 100

will show you the largest files and directories on the / partition.

Anonymous said...

Use the '-n' option to vim (and use 'view' instead to run vim in read-only mode). Then it won't need to make a "swap" file and prepare memory for tracking changes that you make. This makes it fast enough to open large log files like that.

Bur[N]er said...

First thing I always do is "sudo apt-get clean" to dump all the .debs that have been downloaded and are no longer needed since they're already installed.

Mackenzie said...
This comment has been removed by the author.