Assume that we have the following file where each column is separated by a [TAB] character:

Group-ID   Category-ID   Text        Frequency
----------------------------------------------
200        1000          oranges     10
200        900           bananas     5
200        1000          pears       8
200        1000          lemons      10
200        900           figs        4
190        700           grapes      17

I’d like to have this file sorted by these columns and in this specific order (note that column 4 is sorted before column 3 and that column 4 is sorted in reverse order):

• Group ID (integer)
• Category ID (integer)
• Frequency “sorted in reverse order” (integer)
• Text (alpha-numeric)

This should sort the file into this format:

Group-ID   Category-ID   Text        Frequency
----------------------------------------------
190        700           grapes      17
200        900           bananas     5
200        900           figs        4
200        1000          lemons      10
200        1000          oranges     10
200        1000          pears       8

The quick answer is that these sort arguments would solve the problem:

sort -t $'\t' -k 1n,1 -k 2n,2 -k4rn,4 -k3,3 <my-file>

A description of what it all means please read on. The first thing we need to do is to tell sort to use TAB as a column separator (column separated or delimited) which we can do using:

sort -t $'\t' <my-file>

If our input file was comma separated we could have used:

sort -t "," <my-file>

The next step is define that we want the file sorted by columns 1, 2, 4 and 3 and in this particular order. The key argument “-k” allows us to do this. The tricky part is that you have to define the column index twice to limit the sort to any given column, e.g. like this “-k 1,1″. If you only specify it once like this “-k 1″ you’re telling Unix “sort” to sort the file from column 1 and until the end of the line which is not what we want. If you want to sort column 1 and 2 together you’d use “-k 1,2″.  To tell sort to sort multiple columns we have to define the key argument “-k” multiple times. The sort arguments required to sort our file in column order 1, 2, 4 and 3 will therefore look like this:

sort -t $'\t' -k 1,1 -k 2,2 -k 4,4 -k 3,3 <my-file>

We however want the 4th column sorted in reverse order which we can instruct sort to do by changing the argument from “-k 4,4″ to “-k 4r,4″. The “r” option reverses the sort order that column only. There’s only one problem left to solve and that is that sort by default will interpret numbers as text and will sort e.g.  the number 10 ahead of 2. We solve this by adding the “n” option to tell “sort” to sort a column using its numerical values e.g. “-k 1n,1″. Note that the “n” option is only attached to the first number to the left of the comma. Since the 4th column is sorted in both reversed order and using numerical values we can combine the options like this “-k 4rn,4″

So by adding all of these options together with end up with:

sort -t $'\t' -k 1n,1 -k 2n,2 -k 4rn,4 -k 3,3 <my-file>

I hope someone will find this useful. I tested this solution on both Linux and OS X. The documentation for the Unix sort command can be found using your man command “man sort” and “info sort”.

Related posts:

1. How to automatically provide an answer to Unix commands

ls -d $PWD/*

or

ls -ld $PWD/*

No related posts.

Since the “-l” parameter uses Kbit/s the correct value in my case is 500

scp -l 500 big-file.tar someuser@someserver.com:backups/.

Related posts:

1. How to automatically provide an answer to Unix commands

cp: overwrite `destination/./b.txt'?

Many Unix/Linux commands that operate on files may stop and ask for confirmation for each file before completing an action. Of course for many Unix/Linux commands there are parameters that allows you to specify the desired behavior, but there are other commands that doesn’t have that capability. Advanced Unix users just look up the “yes” command on how to solve this. Here’s a contrived example where a directory called “destination” contains two files “a.txt” and “b.txt”. We will now copy two files “b.txt” and “c.txt” from a directory called “source”

|-- destination
|   |-- a.txt
|   `-- b.txt
`-- source
 |-- b.txt
 `-- c.txt

We’ll copy the files using the “cp” command

cp source/*.txt destination/.

When we do this we’ll get the following response on the command line where the OS is asking us what it want us to do since one file being copied already exist in the destination directory i.e. “b.txt”

cp: overwrite `destination/./b.txt'?

Unfortunately on Linux the cp command doesn’t have an option to automatically answer this question with a ‘y’ or ‘n’ answer. There’s more than one solution to this problem depending on what you want to do, and one solution is to use the Unix “yes” command. This command will output a string repeatedly until killed. If we want to avoid overwriting any of the files in the destination directory we could use the “yes” command to answer all questions with a no “n”. Below you can see how the “yes” command is used to automatically provide the answer “n” to the “cp” command. Use “man yes” for more information about the “yes” command.

yes n | cp source/*.txt destination/.

This successfully copies all files with the exception of the “b.txt” file which is what we wanted. The “yes” command automatically provided the answer “n” to the “cp” command question whether or not we wanted to overwrite the destination file.

In the case that you mess up when you use the “yes” command for example by typing this:

yes bingo

The “yes” command will output the word “bingo” repeatedly to the screen. You can stop it by typing Control-C or in a different shell type “pgrep yes” which will display the “yes” commands PID (process ID) which you can use to kill the “yes” command by typing “kill <PID>”

pgrep yes

In my case it returned 24940 so I killed it with:

kill 24940

Related posts:

1. Unix/Linux Sort Multiple Columns, Tab Delimited and Reverse Sort Order
2. How to control or throttle SCP file transfer speed