Just Another ITS Systems Guy

SEVERE: Socket accept failed
 java.net.SocketException: SSL handshake error
 javax.net.ssl.SSLException: No available certificate or key corresponds to the
SSL cipher suites which are
enabled.
        at
 org.apache.tomcat.util.net.jsse.JSSESocketFactory.acceptSocket(JSSESocketFactory.java:150)
        at
 org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:310)
        at java.lang.Thread.run(Unknown Source)

This is the reason you never change anything on Friday at 5pm. While attempting to update the SSL certificate for the MySQL Enterprise Monitor (for which the process has no documentation), I managed to break it in a way that caused a few hundred megs of these errors to dump to the catalina log for MEM. Oh, and it meant no monitoring was taking place for a few minutes.

Sigh.

Lesson re-learned. Glad I made a backup of the keystore before I started mucking with it. Now we wait for MySQL to provide me with the correct documentation (after they write it some time this weekend). You would think someone would have already encountered this with their product considering how long it’s been out there already.

At least we have monitoring back.

So today I found myself staring at the Red Hat LVM documentation wherein I began to silently cuss.  You see, I needed to double the LUN size for MySQL’s usage on one of my servers.  These servers are our first that are directly attached into the SAN and running Linux, so we’re working a bit off the map here.  To make matters more difficult, we’re using multipathd to manage the SAN connections between the system and SAN, so it wasn’t very clear on what exactly I should do in this case.  The LVM docs are somewhat … lacking.

Thanks to the wonder of Google and sheer dumb luck, I ran across this post on one of the RHEL5 mailing lists.  In that case, the author wasn’t sure if it was the right method.  But, I had a secondary system and was willing to run with scissors for a moment since we’re not fully in production yet.

In short, the basic flow of operations is:

1. Figure out the multipath I/O device name.
2. Figure out the underlying device IDs (or device names)
3. Issue the resize of the LUN in your SAN.
4. Tell the kernel to rescan the underlying device IDs so it sees the new LUN size.
5. Tell multipathd that a resize has occurred.
6. Issue a pvresize so LVM knows it has more extents to work with now.
7. Issue an lvresize to increase the logical volume size.
8. Run resize2fs and do an online resize of the filesystem.
9. Make popcorn.
10. Watch a movie.

The first time I attempted the process (in a similar, but not quite fashion), I caused the system hang all LVM commands.  I turned off multipathd thinking that it would need to be off while I did the resize.  This appears to not be a healthy way to do it because I ended up having to warm cycle the system.  This is the point where I stopped reading the LVM documentation and found the mailing list post.    Tried it out and it worked.

So, without further ado …

#  multipath -ll mpath0
mpath0 (360060160dac711004c6fa9d07c7cde11) dm-2 DGC,RAID 10
[size=50G][features=1 queue_if_no_path][hwhandler=1 emc][rw]
\_ round-robin 0 [prio=2][active]
 \_ 1:0:1:0 sdc 8:32  [active][ready]
 \_ 2:0:1:0 sde 8:64  [active][ready]
\_ round-robin 0 [prio=0][enabled]
 \_ 1:0:0:0 sdb 8:16  [active][ready]
 \_ 2:0:0:0 sdd 8:48  [active][ready]

#  for i in `multipath -ll mpath0 | grep sd | awk '{print $2}'`; do
> echo $i ; done
1:0:1:0
2:0:1:0
1:0:0:0
2:0:0:0

#  for i in `multipath -ll mpath0 | grep sd | awk '{print $3}'`; do
> blockdev --rereadpt /dev/$i ; done
BLKRRPART: Input/output error
BLKRRPART: Input/output error

#  multipathd -k"resize multipath mpath0"
ok

#  multipath -ll mpath0
mpath0 (360060160dac711004c6fa9d07c7cde11) dm-2 DGC,RAID 10
[size=100G][features=1 queue_if_no_path][hwhandler=1 emc][rw]
\_ round-robin 0 [prio=2][enabled]
 \_ 1:0:1:0 sdc 8:32  [active][ready]
 \_ 2:0:1:0 sde 8:64  [active][ready]
\_ round-robin 0 [prio=0][enabled]
 \_ 1:0:0:0 sdb 8:16  [active][ready]
 \_ 2:0:0:0 sdd 8:48  [active][ready]

#  pvresize /dev/mapper/mpath0
  Physical volume "/dev/mpath/mpath0" changed
  1 physical volume(s) resized / 0 physical volume(s) not resized

#  lvresize -L 50G /dev/VolGroupMySQL/mysql-san
  Extending logical volume mysql-san to 50.00 GB
  Logical volume mysql-san successfully resized

#  resize2fs /dev/VolGroupMySQL/mysql-san
resize2fs 1.39 (29-May-2006)
Filesystem at /dev/VolGroupMySQL/mysql-san is mounted on /var/lib/mysql; on-line resizing required
Performing an on-line resize of /dev/VolGroupMySQL/mysql-san to 13107200 (4k) blocks.
The filesystem on /dev/VolGroupMySQL/mysql-san is now 13107200 blocks long.

#  df
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      9.7G  3.3G  6.0G  36% /
/dev/sda1             122M   13M  103M  11% /boot
tmpfs                 3.9G     0  3.9G   0% /dev/shm
/dev/mapper/VolGroupMySQL-mysql--san
                       50G  795M   46G   2% /var/lib/mysql

One thing to note is the BLKRRPART errors from blockdev.  This appears to be “normal” as far as I can tell.  The kernel through some log messages (included below), but they appear harmless as far as I can discern.  The SCSI notices occurred when I issued the blockdev command.  The device-mapper multipath warning came from the multipathd resize command.

SCSI device sdc: 209715200 512-byte hdwr sectors (107374 MB)
sdc: Write Protect is off
sdc: Mode Sense: 87 00 00 08
SCSI device sdc: drive cache: write through
sdc: detected capacity change from 53687091200 to 107374182400
 sdc: unknown partition table
SCSI device sde: 209715200 512-byte hdwr sectors (107374 MB)
sde: Write Protect is off
sde: Mode Sense: 87 00 00 08
SCSI device sde: drive cache: write through
sde: detected capacity change from 53687091200 to 107374182400
 sde: unknown partition table
SCSI device sdb: 209715200 512-byte hdwr sectors (107374 MB)
sdb: test WP failed, assume Write Enabled
sdb: asking for cache data failed
sdb: assuming drive cache: write through
sdb: detected capacity change from 53687091200 to 107374182400
 sdb:<6>sd 1:0:0:0: Device not ready: <6>: Current: sense key: Not Ready
    Add. Sense: Logical unit not ready, manual intervention required

end_request: I/O error, dev sdb, sector 0
printk: 68 messages suppressed.

Buffer I/O error on device sdb, logical block 0
 unable to read partition table
SCSI device sdd: 209715200 512-byte hdwr sectors (107374 MB)
sdd: test WP failed, assume Write Enabled
sdd: asking for cache data failed
sdd: assuming drive cache: write through
sdd: detected capacity change from 53687091200 to 107374182400
 sdd:<6>sd 2:0:0:0: Device not ready: <6>: Current: sense key: Not Ready
    Add. Sense: Logical unit not ready, manual intervention required

end_request: I/O error, dev sdd, sector 0
 unable to read partition table
device-mapper: multipath emc: long trespass command will be send
device-mapper: multipath emc: honor reservation bit will not be set (default)
device-mapper: multipath: Using dm hw handler module emc for failover/failback and device management.
device-mapper: multipath emc: emc_pg_init: sending switch-over command

And while I was at it. I figured out I could do something similar with mysql queries. Instead of turning on full Query-logging in mysql (which probably means a restart of a running production mysql) I could just sniff it

tshark -i eth0 -aduration:60 -d tcp.port==3306,mysql -T fields \
   -e mysql.query 'port 3306'

audun.ytterdal.net » Tshark to the rescue

I’ve been using wireshark for awhile now.  I ran across this the other day while trying to debug a non-MySQL issue.  It was a good reminder of wireshark’s ability to do live dissection of protocol from the command line.  I’ve had to do this before while trying to debug NFS issues (specifically, trying to work backwards from a packet to figure out the name of an open file descriptor from a file handle and fsid).  Wireshark’s ability to do this on the fly and from the command line is pretty powerful.

Right now I have a need to figure out why MySQL is showing a bunch of aborted connections.  I was hoping that mysql.error would give me something intelligent to poke after, but I think this occurs all within the MySQL instance.  More digging to do.  It’s good to know that I can do some impromptu poking around of the MySQL packet if I without firing up a full wireshark session on the server if I need to.

If you have a need to do this, these two things will be helpful.

• Wireshark Display Filter reference for the MySQL Protocol
• MySQL Internals ClientServer Protocol

Happy sniffing!

• Electric Alchemy: Cracking Passwords in the Cloud: Breaking PGP on EC2 with EDPR.  An interesting discussion about using Amazon EC2 to make it possible for the underfunded layman to crack passwords.
• “There is no YEAR ZERO for HTML. It’s a conversation [...]“. One of the comments on Mark Pilgrim’s post about Why do we have an IMG element? in HTML. Short but good history lesson on how we got here from there.
• Innovating Cron: Announcing Norc Do we really need a new cron-like tool? Maybe we do. The Hacker News discussion has some useful bits of information on queuing and batch systems too.
• Trax STI. Suburu based snowcat. WANT.

If the only reason you’re only wearing one hat is because you’ve always only worn one hat, that’s not a good reason.

Seth’s Blog: Ms. In-between.

Tradition for tradition’s sake is never a good reason to continue doing something in a technical environment.  The landscape is ever changing.  You should always be re-evaluating what you’re doing while keeping an eye on the bleeding edge of your profession. It’s the difference between being relevant and being the (only) caretaker of a legacy system.

When was the last time you challenged a tradition?

The programmer-managed infrastructure suffers from a death by a thousand cuts. The programmer is competent with technology and fully capable of setting up a system that can support the application being built. The programmer, however, lacks a detailed understanding of ongoing infrastructure management. Consequently, the programmer-managed infrastructure ultimately leads to an environment incapable of adjusting to changing demands and potentially opens vulnerabilities to hackers through discreet channels.

The reverse is true of the sys admins who fancy themselves programmers. They can craft Perl programs to do just about any task. Those programs, however, ultimately lack the solid architecture that programming skills provide.

Your Cloud Needs a Sys Admin - O’Reilly Broadcast.

Fair warning:  I’m a sysadmin so my opinion might be slightly biased on this idea.

Yes, you do need a sysadmin.  Just like the blog posts suggests.  Like everything in business, you want to use the right tool for the right job.  Sure, you can use a wrench to hammer a nail in, but in the process you’re likely going to smash a finger, chip the wood, and spend excessive time frustrating yourself when pounding that nail into place doesn’t go fast enough to meet your nail-hammerin’ schedule.

Sysadmins are hammers (the right tool) when it comes to nails (managing systems and infrastructures).  A good sysadmin, like the blog post alludes, has a broad and necessary knowledge required for running a system (or set of systems) effectively and safely (read:  securely).  More often than not, we have the experience needed to tell you when exactly you can cut corners, when you shouldn’t, and why doing so may or may not be the right thing for your environment.

There’s an old AI koan that I’m reminded of.

A novice was trying to fix a broken Lisp machine by turning the power off and on. Knight, seeing what the student was doing spoke sternly: “You can not fix a machine by just power-cycling it with no understanding of what is going wrong.” Knight turned the machine off and on. The machine worked.

Ubuntu Linux adds private cloud backing

Canonical’s upcoming server upgrade supports the Eucalyptus project’s open source system for cloud implementation using hardware and software already in place

Canonical is touting private cloud capabilities in an upgrade to its Ubuntu Linux OS being announced on Tuesday.

Available for free download on October 29, Ubuntu 9.10 Server Edition introduces UEC (Ubuntu Enterprise Cloud), an open source cloud computing environment based on the same APIs as Amazon EC2 (Elastic Compute Cloud). Businesses can take advantage of private clouds, Canonical said.

Ubuntu Linux adds private cloud backing | Open Source - InfoWorld.

This should prove interesting.  If we were able to leverage something like this, we could build out a private cloud for researchers.  The Eucalyptus system certainly looks useful.  Especially if they’re touting it as API compatible with other external cloud vendors.  We’d certainly need to do some heavy investigation to figure out what running our own cloud would actually mean.  I can certainly see it as being completely different than running a classic high performance computing grid.

You, too, can have a cloud in the privacy of your own home!  Time to keep up with the Jones’s again!

• Larry Ellison hates cloud computing - funny clip of Ellison lambasting the idea of clouds. Yes, really, clouds have been around for over a decade, we just didn’t know it (or realize it).

It doesn’t have to be a wise decision or a perfect one. Just make one.

In fact, make several. Make more decisions could be your three word mantra.

Seth’s Blog: Make a decision.

Finally, someone agrees with me.  One of the biggest demoralizers in an organization (in my experience) was the lack of someone standing up and just making a decision.  Pick a direction.  Any direction.  It may not be the right decision (and may even turn out to be flat wrong).  But, it’s a direction that your employees can latch onto and do something with.

The lack of decisions mean no traction, no effort, no movement.  Decisions mean progress.

You can always correct a bad decision.  You can’t always recover from the lack of one.

• Your People - Rand in Repose: “Your People will piss you off because the relationship is genuine. They do not coddle and they do not spin. Consequently, Your People error-correct you in ways that others cannot.”
• Wordle: ” Wordle is a toy for generating “word clouds” from text that you provide.” This looks like a great way to make designs for tshirts or posters. Imagine doing these for your organization, covering all the words that describe your mission statement.
• justniffer: a packet sniffer that outputs common log format style information on your packets. Nifty!
• piwik: Someone’s gone and done it … an open source clone of Google Analytics.