A Scala script for generating summary statistics and frequency distributions from a log file

Git SVN Workflow

Fetch and import a Subversion repository into a local Git repository

git svn clone -s <svn repo> ./

Create and checkout a local topic branch to work on called task-1.

git checkout -b task-1

Create/delete/modify files then add any untracked files in your work directory to the Git index

git add .

Record all changes to the Git repository

git commit -a -m "Insert commit message here..."

Update master (local Subversion tracking branch) with any changes from Subversion

git checkout master
git svn rebase

Update task-1 with any changes to local master

git checkout task-1
git rebase -i master

To see any conflicts type

git diff

Fix conflicts by editing the files by hand or run git mergetool

Add the merged file(s) and finish the rebase

git add .
git rebase --continue

Merge changes from task-1 to local Master

git checkout master
git merge --ff-only task-1

Commit the changes from local Master to Subversion

git svn dcommit

SSH into an Amazon EC2 Instance

To start an EC2 instance and SSH (Secure Shell) to the instance we need to use an RSA key pair.

Configure the EC2 firewall

First we need to open a port on the EC2 firewall, for SSH communication, and limit access to the public IP address of our client computer.

To find the client's IP address enter

curl jsonip.com

Using a security group of trader, the defauly SSH port of 22, and a client IP address of 111.40.27.251, we'll open the port using the EC2 command line tool

ec2-authorize trader -p 22 -s <your public IP address>

Generate the Key Pair

You can create your own key pair and upload the public key to Amazon using ec2-import-keypair , or Amazon can generate the key pair for you using ec2-create-keypair.

A benefit of using your own keys is that you can use this same key across regions, with the Amazon generated key it's only valid for the region used when the key was generated.

When you create a new EC2 instance Amazon will append the public key to /home/<user>/.ssh/authorized_keys

We'll generate the keys using both methods.

First create a directory where we will store the private key

mkdir ~/.ec2

Amazon Key Pair generation

The following ec2-create-keypair command will create a new RSA key pair with the name ec2-keypair. Amazon will store the public key and the command will output the private key to the console.

ec2-create-keypair ec2-keypair

Outputs to your console the private key and associated details

KEYPAIR 
ec2-keypair1f:51:ae:28:bf:89:e9:d8:1f:25:5d:37:2d:7d:b8:ca:9f:f5:f1:6f
-----BEGIN RSA PRIVATE KEY-----
MIIEoQIBAAKCAQBuLFg5ujHrtm1jnutSuoO8Xe56LlT+HM8v/xkaa39EstM3/aFxTHgElQiJLChp
HungXQ29VTc8rc1bW0lkdi23OH5eqkMHGhvEwqa0HWASUMll4o3o/IX+0f2UcPoKCOVUR+jx71Sg
5AU52EQfanIn3ZQ8lFW7Edp5a3q4DhjGlUKToHVbicL5E+g45zfB95wIyywWZfeW/UUF3LpGZyq/
ebIUlq1qTbHkLbCC2r7RTn8vpQWp47BGVYGtGSBMpTRP5hnbzzuqj3itkiLHjU39S2sJCJ0TrJx5dummy
-----END RSA PRIVATE KEY-----

Now we will copy the private key to ~/.ec2/ec2-keypair.pem

echo "-----BEGIN RSA PRIVATE KEY-----
MIIEoQIBAAKCAQBuLFg5ujHrtm1jnutSuoO8Xe56LlT+HM8v/xkaa39EstM3/aFxTHgElQiJLChp
HungXQ29VTc8rc1bW0lkdi23OH5eqkMHGhvEwqa0HWASUMll4o3o/IX+0f2UcPoKCOVUR+jx71Sg
5AU52EQfanIn3ZQ8lFW7Edp5a3q4DhjGlUKToHVbicL5E+g45zfB95wIyywWZfeW/UUF3LpGZyq/
ebIUlq1qTbHkLbCC2r7RTn8vpQWp47BGVYGtGSBMpTRP5hnbzzuqj3itkiLHjU39S2sJCJ0TrJx5dummy
-----END RSA PRIVATE KEY-----" > ~/.ec2/ec2-keypair.pem

User Key Pair generation

ssh-keygen -t rsa

Enter file in which to save the key ( /home/<user>/.ssh/id_rsa): /home/<user>/.ec2/ec2-keypair.pem

Skip the passphrase

Now copy the public key to Amazon

ec2-import-keypair ec2-keypair --public-key-file ~/.ec2/ec2-keypair.pub

Set private key permissions

Set the permissions so that you have read, write and execute permissions on the directory where the private key is kept.

chmod 700 ~/.ec2

Set the permissions so that you have read and write permissions on the private key

chmod 600 ~/.ec2/ec2-keypair.pem

Set environment variables

Export the name of the keypair, not the filename. So ec2-keypair, not ec2-keypair.pem

export EC2_KEYPAIR_NAME=~/.ec2/ec2-keypair

Create the EC2 instance

ec2-run-instances ami-xxxxx -g default -k $EC2_KEYPAIR_NAME -t m1.small;

SSH to the instance

We can now SSH to the instance by providing our private key (identity file)

ssh -i ~/.ec2/ec2-keypair.pem ubuntu@ec2-xxx-xx-xx-xx.compute-1.amazonaws.com

An alternative to providing the identity file each time is to add the identity file in your ~/.ssh/config file

Host ec2-*.amazonaws.com 
     IdentityFile ~/.ec2/ec2-keypair.pem

So now when you SSH to any Amazon EC2 server simply enter the following and the identity file is automatically provided.

ssh ubuntu@ec2-xxx-xx-xx-xx.compute-1.amazonaws.com

Prevent SSH timeouts

To stop the SSH connection from timing out see this previous post

Terminate the instance

You are billed while the instance is running so don't forget to terminate it

ec2-terminate-instances <instance_id>

The above was tested using:

Server
- Ubuntu 10.04 Lucid Lynx

Client
- Ubuntu 11.04 Natty Narwhal

References:
- https://help.ubuntu.com/community/SSH/OpenSSH/Keys
- https://help.ubuntu.com/community/EC2StartersGuide
- http://www.symantec.com/connect/articles/ssh-user-identities

Keeping SSH connections alive

To stop a SSH connection from timing out due to inactivity, we can configure the server or the client, to send a regular keep alive message.

For an explanation of the following ssh_config options see http://linux.die.net/man/5/ssh_config

Configuring the server

Edit /etc/ssh/ssh_config

sudo vim /etc/ssh/ssh_config

Add the following lines

ClientAliveInterval 15
ClientAliveCountMax 4

And restart the SSH server for the changes to take affect

/etc/init.d/ssh restart

Configuring the client

Edit vim ~/.ssh/config

sudo vim ~/.ssh/config

Add the following lines so connections to all Hosts have the same SSH options

Host *
    ServerAliveInterval 15
    ServerAliveCountMax 4

Security consideration

Keeping a SSH connection permanently open could be a security issue.

To explicitly close an inactive connection after 60 seconds set the ServerAliveInterval/ClientAliveInterval to 60, and the ServerAliveCountMax/ClientAliveCountMax to 0

*AliveInterval 60
*AliveCountMax 0

The above was tested using:

Server
- Ubuntu 10.04 Lucid Lynx

Client
- Ubuntu 11.04 Natty Narwhal

PostgreSQL hanging after a restart

If PostgreSQL behaves strangely after a restart, e.g. SQL statements appear to hang or a table can't be truncated/dropped, chances are the database didn't close down cleanly and a transaction wasn't committed/rolled back.

If there are any transactions in the view view-pg-prepared-xacts roll them back and the problem will hopefully be resolved.

First login to Postgres

PGPASSWORD=password psql -d trade -U tradeadmin

See if there are any outstanding transactions

SELECT database, gid FROM pg_prepared_xacts;

database | gid
---------------+------------------------------------------------------------------------------
trade | 1096044365_bmV0cGFnZWNvbW1vbi50bTAwMDAxMDAwMDQ=_bmV0cGFnZWNvbW1vbi50bTE2OQ==
(1 row)

To rollbank the transaction enter ROLLBACK PREPARED and the gid .e.g.

ROLLBACK PREPARED '1096044365_bmV0cGFnZWNvbW1vbi50bTAwMDAxMDAwMDQ=_bmV0cGFnZWNvbW1vbi50bTE2OQ==';

Done.

The above was tested using:

Server
- PostgreSQL 8.4
- Ubuntu 10.04 Lucid Lynx


References: 
- http://www.postgresql.org/docs/8.4/static/view-pg-prepared-xacts.html
- http://www.postgresql.org/docs/8.4/static/sql-rollback-prepared.html
- https://doc.nuxeo.com/display/KB/I+can't+delete+my+PostgreSQL+database

Remote JVM profiling on Amazon EC2

Using SSH Port Forwarding (SSH tunneling) establish a secure connection with an Amazon EC2 instance, run Java VisualVM or JConsole locally, and monitor a remote Tomcat process over this connection.

This approach requires Tomcat 6.0.24 or latter as this was the version that the JmxRemoteLifecycleListener was introduced. I'll look at how to monitor older versions of Tomcat using X11 Forwarding in an upcomming post.

Setup security

First we need to open a port on the EC2 firewall, for JMX/RMI communication, and limit access to the public IP address of our client computer.

To find the client's IP address enter

curl jsonip.com

Using a security group of trader, a port of 10001 for JMX/RMI, and a client IP address of 111.40.27.251, we'll open the port using the EC2 command line tool

ec2-authorize trader -p 10001 -s <your public IP address>

Configure Tomcat

Determine the host you want to monitor. 16 means filter only running instances.

ec2-describe-instances --filter instance-state-code=16

Now SSH to the instance

ssh -i $EC2_AWS/ec2.pem neil@ec2-123-12-1-123.compute-1.amazonaws.com

To check what version of Tomcat is running

$CATALINA_HOME/bin/version.sh

To set the JMX/RMI port number we need to enable and configure the JMX Remote Lifecycle Listener for Tomcat. First we need to download the required jar.

sudo wget -P $CATALINA_HOME/lib http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.28/bin/extras/catalina-jmx-remote.jar

Install into Tomcat

cp catalina-jmx-remote.jar $CATALINA_HOME/lib

Next edit /etc/default/tomcat6

sudo vim /etc/tomcat6/server.xml

And add the following listener

<Listener className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener" rmiRegistryPortPlatform="10001" rmiServerPortPlatform="10001"/>

Now edit the /etc/default/tomcat6 file

sudo vim /etc/default/tomcat6

And add the following system properties. Note that the java.rmi.server.hostname is the public hostname of the EC2 instance.

JAVA_OPTS="${JAVA_OPTS} -Djava.rmi.server.hostname=ec2-123-12-1-123.compute-1.amazonaws.com -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"

Restart Tomcat for the changes to take affect

sudo /etc/init.d/tomcat6 restart

Monitor Tomcat

Now, on the client computer create the SSH tunnel that will forward port 10001 on our local client computer to port 10001 on the remote EC2 instance.

ssh -N -v -L 10001:localhost:10001 ubuntu@ec2-123-12-1-123.compute-1.amazonaws.com

Open a new terminal and enter either

$JAVA_HOME/bin/jvisualvm -J-Djava.io.tmpdir=/tmp/tomcat6-tmp

Or

$JAVA_HOME/bin/jconsole -J-Djava.io.tmpdir=/tmp/tomcat6-tmp localhost:10001

Sometimes takes a few seconds for the localhost:10001 connection to show in the UI.

Done.

The above was tested using:

Server
- Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
- Tomcat 6.0.28
- Ubuntu 10.04 Lucid Lynx

Client
- Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
- Ubuntu 11.04 Natty Narwhal

Why are we using -J-Djava.io.tmpdir=/tmp/tomcat6-tmp?

See http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7009828

References:
- http://download.oracle.com/javase/6/docs/technotes/guides/visualvm/intro.html
- http://tomcat.apache.org/tomcat-7.0-doc/config/listeners.html
- http://gabenell.blogspot.com/2010/04/connecting-to-jmx-on-tomcat-6-through.html