This script is designed to show a number of recent commits to the users current SVN branch in an easy readable format.
Showing posts with label Linux. Show all posts
Showing posts with label Linux. Show all posts
Sunday, 10 August 2014
Host Information
This script is designed to retrieve host information from the kernel to the memory. It assigns standard commands to variables and then echo's everything out in a nice readable form.
IP Telnet
This script is your basic Telnet script.
In order to use this script you must create a text file called 'ip' and fill it with the ip(s) you wish to connect to.
In order to use this script you must create a text file called 'ip' and fill it with the ip(s) you wish to connect to.
Pack Up Tar Files Script
This script is used to pack up files into a "tar.gz", this can easily be change to a "tar.xz" by replacing all instances of "tar.gz".
Watcher Script
You just need to set this script to run on startup and there you go, if anyone tries to connected to your pc via ssh,
Chmod Examples
chmod [OPTION]... MODE[,MODE]... FILE...
Chmod (Change Mode) is used to change the permissions of a file, this is needed if you want to create shell scripts or set the permissions of a file.
Chmod (Change Mode) is used to change the permissions of a file, this is needed if you want to create shell scripts or set the permissions of a file.
Wednesday, 29 January 2014
What are Index Nodes?
I'm writing about this because it came up as an issue while at work one day.
Background info!
Quite a while ago I wrote some software which keeps an eye on a number of servers and sends me an email if anything goes wrong, one of the things being if the servers hard disk becomes full / the system cannot create new files.
So I'm sat doing my daily java programming and I receive and email saying our main server is full, pretty strange because each server should have at least 20GB of space free at all times. So I log on to the system to see whats taking up all this room, the first command I do is df -h and I see the following output:
This is not the output I expected, so I thought maybe there was a problem writing files to the disk so I run a touch text.txt command and this seems to run perfectly fine. After looking thought my email software to see if it had somehow made a mistake I get another email saying the server is having problems writing logs, this is when I start getting confused.
I log back onto the system and check the logs. The logs had indeed stopped, as a test I wrote a small bash script to create a new file, it didn't work, instead it came back saying the hard disk was full.
After some extreme head scratching and chin stoking I found the source of my problem by running the command df -i and seeing that my IUse% was 100%.
The problem was these strange little things called 'i-nodes'.
Index Nodes (i-nodes)
An index node is a data-structure used to represent a file system object, this can be disk block location, creation time and a whole host of other metadata. Each index node is roughly 128 bytes in size and all index nodes should really only ever take up 1% of your hard disk.
So for each file on your hard disk there is a i-node. So the solution presented itself, All I had to do was find the largest directory and backup and remove the files. To find the largest directory on the system I ran this command: du -a | sort -n -r and I instantly found the issue. Another developer had written a program which was producing a hell of a lot of log files, every 2 seconds a new log file (very small in size) was created. While this wasn't taking up much room on the hard drive it was taking up alot of index nodes.
So there's a solution to anyone else having this issue.
Background info!
Quite a while ago I wrote some software which keeps an eye on a number of servers and sends me an email if anything goes wrong, one of the things being if the servers hard disk becomes full / the system cannot create new files.
So I'm sat doing my daily java programming and I receive and email saying our main server is full, pretty strange because each server should have at least 20GB of space free at all times. So I log on to the system to see whats taking up all this room, the first command I do is df -h and I see the following output:
This is not the output I expected, so I thought maybe there was a problem writing files to the disk so I run a touch text.txt command and this seems to run perfectly fine. After looking thought my email software to see if it had somehow made a mistake I get another email saying the server is having problems writing logs, this is when I start getting confused.
I log back onto the system and check the logs. The logs had indeed stopped, as a test I wrote a small bash script to create a new file, it didn't work, instead it came back saying the hard disk was full.
After some extreme head scratching and chin stoking I found the source of my problem by running the command df -i and seeing that my IUse% was 100%.
The problem was these strange little things called 'i-nodes'.
Index Nodes (i-nodes)
An index node is a data-structure used to represent a file system object, this can be disk block location, creation time and a whole host of other metadata. Each index node is roughly 128 bytes in size and all index nodes should really only ever take up 1% of your hard disk.
So for each file on your hard disk there is a i-node. So the solution presented itself, All I had to do was find the largest directory and backup and remove the files. To find the largest directory on the system I ran this command: du -a | sort -n -r and I instantly found the issue. Another developer had written a program which was producing a hell of a lot of log files, every 2 seconds a new log file (very small in size) was created. While this wasn't taking up much room on the hard drive it was taking up alot of index nodes.
So there's a solution to anyone else having this issue.
Friday, 15 November 2013
What The Hell is A Shell?!
I've had a lot of people ask this question before, myself included. I thought I would write a small straight to the point post about the differences between the most common.
I'll start by explaining exactly what a shell is and what it is used for. (Although most of you will probably already know)
What the hell is a shell?!
Before graphical user interfaces the command line was the main interface for users, just like now where we have KDE, Gnome etc... the command line had shells. Shells were used to give the user a slightly different environment in terms of functionality and scripting. In technical terms a shell is a command line interpreter developed to make things easier for users. One example of this would be filename/command completion (do this by pressing tab after the first part of the command/filename). These small features that a lot of people see as part of linux is actully part of the shell.
Popular shells
The most popular shell (and the shell most of you will probably be using) is the Bash Shell written by a guy called Brian Fox. This is the most popular shell due to its robustness and tons of features. I use this shell at home all the time for scripting.
The Bourne Shell, AKA 'sh'. This shell was developed by a guy called Stephen Bourne, It quickly became the default shell for Unix due to its support for variables, piping, looping and a verity of other usefull stuff. I use this shell mainly when writing software for embedded systems as it is the default shell.
The C-Shell AKA 'csh' was developed by a guy called Bill Joy. This guy added useful things like aliases (short cuts for long commands) and command history. This shell was modeled on the C programming language, this was a good thing as Unix was written in C...although it did mean users of the bourne shell had to learn C in order to use the C-Shell.
Which is the best shell?
There are a large number of shells to choose from, I've only named three here as these are the most popular. I would say the best shell is the Bash Shell due to how popular it is, this means most scripts written with this shell will most likely work on any machine.
I'll start by explaining exactly what a shell is and what it is used for. (Although most of you will probably already know)
What the hell is a shell?!
Before graphical user interfaces the command line was the main interface for users, just like now where we have KDE, Gnome etc... the command line had shells. Shells were used to give the user a slightly different environment in terms of functionality and scripting. In technical terms a shell is a command line interpreter developed to make things easier for users. One example of this would be filename/command completion (do this by pressing tab after the first part of the command/filename). These small features that a lot of people see as part of linux is actully part of the shell.
Popular shells
The most popular shell (and the shell most of you will probably be using) is the Bash Shell written by a guy called Brian Fox. This is the most popular shell due to its robustness and tons of features. I use this shell at home all the time for scripting.
The Bourne Shell, AKA 'sh'. This shell was developed by a guy called Stephen Bourne, It quickly became the default shell for Unix due to its support for variables, piping, looping and a verity of other usefull stuff. I use this shell mainly when writing software for embedded systems as it is the default shell.
The C-Shell AKA 'csh' was developed by a guy called Bill Joy. This guy added useful things like aliases (short cuts for long commands) and command history. This shell was modeled on the C programming language, this was a good thing as Unix was written in C...although it did mean users of the bourne shell had to learn C in order to use the C-Shell.
Which is the best shell?
There are a large number of shells to choose from, I've only named three here as these are the most popular. I would say the best shell is the Bash Shell due to how popular it is, this means most scripts written with this shell will most likely work on any machine.
Friday, 3 August 2012
Sed command examples
A command I use constantly at work is the 'sed' command. This command is used when you want to make changes to a file automatically. Most people who use the 'sed' command use it to replace text (which it does brilliantly) but there is so much more
I'm going to show you a few examples of the awesomeness of this command, if you have never used it before I guaranty you will use it after reading this post.
Lets say we have a file which contains the following text:
"This is my favourite place to learn about Linux. Linux is an opensource OS. Linux has a variety of different flavours."
Search and Replace
Using the 'sed' command we can search and replace. for example if I wanted to replace the word 'Linux' with the word 'Unix' I would do the following:
sed 's/Linux/Unix/' aFile.txt
This would change the first occurrence of "Linux" to "Unix" as shown below:
"This is my favourite place to learn about Unix. Linux is an opensource OS. Linux has a variety of different flavours."
Replacing the nth occurrence of a word is just as easy:
sed 's/Linux/Unix/2' aFile.txt
This will skip the first occurrence of "Linux" and replace the second one with "Unix".
"This is my favourite place to learn about Linux. Unix is an opensource OS. Linux has a variety of different flavours."
If we would like to replace all occurrences of a word then we use the "/g" (Global) option:
sed 's/Linux/Unix/g' aFile.txt
All occurrences of the word "Linux" have now been replaced with "Unix"
"This is my favourite place to learn about Unix. Unix is an opensource OS. Unix has a variety of different flavours."
We can mess about with these options and merge them together to replace all occurrences of a word from a certain point, for example replacing all occurrences of the word "Linux" with the word "Unix" after the first occurrence (the example below will make this a bit clearer):
sed 's/Linux/Unix/2g' aFile.txt
So after the second occurrence the word "Linux" changes to "Unix".
Add a line after or before a match
Here we can add a line after a match is found. For this we use the "a" command, for example:
sed '/flavours/ a "a new line!"' aFile.txt
and the result is as follows:
"This is my favourite place to learn about Linux. Unix is an opensource OS. Linux has a variety of different flavours.
a new line!"
The same thing can work adding a line before a match, we do this using the "i" command.
sed '/flavours/ i "a new line!"' aFile.txt
and the result is as follows:
"This is my favourite place to learn about Linux. Unix is an opensource OS. Linux has a
a new line!
variety of different flavours."
Replace a line
We can search for a match and then replace that whole line using the 'c' command. It pretty much the same format as before:
sed '/flavours/ c "a replaced line!"' aFile.txt
and the result is as follows:
"This is my favourite place to learn about Linux. Unix is an opensource OS. Linux has a
a replaced line!"
The last thing I'm going to show you is the transform option. This is used to transform lower case letters to upper case letters. We use the 'y' command for this. In this example I'm going to change all the small l's and v's to large l's and v's, and vice versa:
sed 'y/lv/LV/ aFile.txt
"This is my faVourite pLace to Learn about linux. Unix is an opensource OS. linux has a Variety of different fLaVours."
I hope this helps you guy's with the 'sed' command, it can be very useful once you get to know it. Any questions are welcome.
I'm going to show you a few examples of the awesomeness of this command, if you have never used it before I guaranty you will use it after reading this post.
Lets say we have a file which contains the following text:
"This is my favourite place to learn about Linux. Linux is an opensource OS. Linux has a variety of different flavours."
Search and Replace
Using the 'sed' command we can search and replace. for example if I wanted to replace the word 'Linux' with the word 'Unix' I would do the following:
sed 's/Linux/Unix/' aFile.txt
This would change the first occurrence of "Linux" to "Unix" as shown below:
"This is my favourite place to learn about Unix. Linux is an opensource OS. Linux has a variety of different flavours."
Replacing the nth occurrence of a word is just as easy:
sed 's/Linux/Unix/2' aFile.txt
This will skip the first occurrence of "Linux" and replace the second one with "Unix".
"This is my favourite place to learn about Linux. Unix is an opensource OS. Linux has a variety of different flavours."
If we would like to replace all occurrences of a word then we use the "/g" (Global) option:
sed 's/Linux/Unix/g' aFile.txt
All occurrences of the word "Linux" have now been replaced with "Unix"
"This is my favourite place to learn about Unix. Unix is an opensource OS. Unix has a variety of different flavours."
We can mess about with these options and merge them together to replace all occurrences of a word from a certain point, for example replacing all occurrences of the word "Linux" with the word "Unix" after the first occurrence (the example below will make this a bit clearer):
sed 's/Linux/Unix/2g' aFile.txt
So after the second occurrence the word "Linux" changes to "Unix".
Add a line after or before a match
Here we can add a line after a match is found. For this we use the "a" command, for example:
sed '/flavours/ a "a new line!"' aFile.txt
and the result is as follows:
"This is my favourite place to learn about Linux. Unix is an opensource OS. Linux has a variety of different flavours.
a new line!"
The same thing can work adding a line before a match, we do this using the "i" command.
sed '/flavours/ i "a new line!"' aFile.txt
and the result is as follows:
"This is my favourite place to learn about Linux. Unix is an opensource OS. Linux has a
a new line!
variety of different flavours."
Replace a line
We can search for a match and then replace that whole line using the 'c' command. It pretty much the same format as before:
sed '/flavours/ c "a replaced line!"' aFile.txt
and the result is as follows:
"This is my favourite place to learn about Linux. Unix is an opensource OS. Linux has a
a replaced line!"
The last thing I'm going to show you is the transform option. This is used to transform lower case letters to upper case letters. We use the 'y' command for this. In this example I'm going to change all the small l's and v's to large l's and v's, and vice versa:
sed 'y/lv/LV/ aFile.txt
"This is my faVourite pLace to Learn about linux. Unix is an opensource OS. linux has a Variety of different fLaVours."
I hope this helps you guy's with the 'sed' command, it can be very useful once you get to know it. Any questions are welcome.
Wednesday, 20 June 2012
How To Install Software On Linux (It's not that hard!)
OK, so I have had enough with people asking me how to install stuff on Linux, it's really not that difficult but it does seem to put a lot of people off the OS. I'm going to go through a few things here such as rpm, tar.gz and tar.bz2.
RPM
RPM stands for "Red hat Package Manager". Files with this extension are installed and managed with the rpm program which I will take you through now.
The rpm program not only installs the software but it checks your system has all the prerequisites. For example if you are trying to install a package called "LinuxWow" but that package requires "LinuxFTW" that you haven't installed rpm will not install "LinuxWow". However if you have "LinuxFTW" then the package will be install as it should. This can save a lot of frustration... believe me.
before installing an rpm file it might be a good idea to check if you already have this installed, to do this run the following command:
rpm -q package_name
-q = query
The commands you would use to install, update or remove a .rpm file are as follows:
Install one or multiple rpm files
rmp -ivh packages1.rpm packages2.rpm
-i = install
-v = verbose information (additional info)
-h = Print 50 hash marks as the package archive is unpacked
Update one or multiple rpm files
rpm -Fvh packages1.rpm packages2.rpm
-F = Freshen
Remove one or multiple rpm files
rmp -e packages1.rpm packages2.rpm
-e = erase.
TAR.GZ & TAR.BZ2
tar files usually contain source code so the idea is that you compile (build) them yourself rather then relying on a program like rpm.
I'll show you how to install these files in 3 steps:
1. First this is to list the packages in the file, this is to make sure we don't over right anything precious on our system. We will use the 'less' command for this. The '|' can be used as an extension of a command, so we could do ' | grep "fileName"' if we wanted to grep the package for a certain file name.
tar tvzf package.tar.gz | less
or
tar tvzf package.tar.bz2 | less
-t = list, this is to list the contents of the package
-v = verbose, you know this by now
-z = this filters the package through gzip, like win-zip if you're from windows
-f = uses the package file
2. If we are happy with everything then we extract the files to a new directory, as shown below:
we make a new directory called newdir:
mkdir newdir
Enter the directory:
cd newdir
and extract the files:
tar xvzf <path>/package.tar.gz
or
tar xvzf <path>/package.tar.bz2
3. once extracted look for a file called 'install' or 'readme' these will give you the rest of the information you need. They usually tell you to run a script called 'configure', after that we run the 'make', (this will ask us to enter our root password) command and then finally the 'make install'.
./configure
make
su
(enter password)
make install
The 'readme' file may contain some other information about the configure script which may require you adding some options but the file should give you all the information you need.
It's really not that difficult to install software on a Linux machine. If you use Ubuntu then a simple "apt-get install package" would suffice. I hope this post makes people realise how easy it is.
RPM
RPM stands for "Red hat Package Manager". Files with this extension are installed and managed with the rpm program which I will take you through now.
The rpm program not only installs the software but it checks your system has all the prerequisites. For example if you are trying to install a package called "LinuxWow" but that package requires "LinuxFTW" that you haven't installed rpm will not install "LinuxWow". However if you have "LinuxFTW" then the package will be install as it should. This can save a lot of frustration... believe me.
before installing an rpm file it might be a good idea to check if you already have this installed, to do this run the following command:
rpm -q package_name
-q = query
The commands you would use to install, update or remove a .rpm file are as follows:
Install one or multiple rpm files
rmp -ivh packages1.rpm packages2.rpm
-i = install
-v = verbose information (additional info)
-h = Print 50 hash marks as the package archive is unpacked
Update one or multiple rpm files
rpm -Fvh packages1.rpm packages2.rpm
-F = Freshen
Remove one or multiple rpm files
rmp -e packages1.rpm packages2.rpm
-e = erase.
TAR.GZ & TAR.BZ2
tar files usually contain source code so the idea is that you compile (build) them yourself rather then relying on a program like rpm.
I'll show you how to install these files in 3 steps:
1. First this is to list the packages in the file, this is to make sure we don't over right anything precious on our system. We will use the 'less' command for this. The '|' can be used as an extension of a command, so we could do ' | grep "fileName"' if we wanted to grep the package for a certain file name.
tar tvzf package.tar.gz | less
or
tar tvzf package.tar.bz2 | less
-t = list, this is to list the contents of the package
-v = verbose, you know this by now
-z = this filters the package through gzip, like win-zip if you're from windows
-f = uses the package file
2. If we are happy with everything then we extract the files to a new directory, as shown below:
we make a new directory called newdir:
mkdir newdir
Enter the directory:
cd newdir
and extract the files:
tar xvzf <path>/package.tar.gz
or
tar xvzf <path>/package.tar.bz2
3. once extracted look for a file called 'install' or 'readme' these will give you the rest of the information you need. They usually tell you to run a script called 'configure', after that we run the 'make', (this will ask us to enter our root password) command and then finally the 'make install'.
./configure
make
su
(enter password)
make install
The 'readme' file may contain some other information about the configure script which may require you adding some options but the file should give you all the information you need.
It's really not that difficult to install software on a Linux machine. If you use Ubuntu then a simple "apt-get install package" would suffice. I hope this post makes people realise how easy it is.
Friday, 11 May 2012
Slightly more advanced guide to Shell Scripting
Hey guys,
In this post I hope to go through a few more advanced options when writing scripts. I'm just going to cover things like methods, parameters and a few other tricks which can make your script look pretty cool.
Methods
As a Java developer by trade I use methods quite a lot, programming without them is sort of odd, for those who don't know a 'method' or 'function' in basic terms is a section of code which can be called at a later time, you can probably see why this would be useful.
Say you have a menu system in your script where the user must choose an option for the script to continue, each option would produce a different outcome, this is a good example of when you would want to use methods.
Examples
#!/bin/bash
echo "***************EXAMPLE SCRIPT***************"
echo "Please choose and option:"
echo " 1 = option A"
echo " 2 = option B "
echo " 3 = option C "
read option
if [[ "$option" == "1" ]]
then
OPTIONA
exit 0
fi
if [[ "$option" == "2" ]]
then
OPTIONB
exit 0
fi
if [[ "$option" == "3" ]]
then
OPTIONC
exit 0
fi
OPTIONA () {
echo "Hello world, this is option A"
}
OPTIONB () {
echo "Hello world, this is option B"
}
OPTIONC () {
echo "Hello world, this is option C"
}
Ok so I'm going to dissect this and explain what each part is doing, its quite simple really.
echo "***************EXAMPLE SCRIPT***************"
echo "Please choose and option:"
echo " 1 = option A"
echo " 2 = option B "
echo " 3 = option C "
This part is the actual menu which will be shown to the user, we use the 'read option' to read the input from the user and assign it to the variable 'option', this is so we can use it later in the if statements.
if [ "$option" == "1" ]
then
OPTIONA
exit 0
fi
if [ "$option" == "2" ]
then
OPTIONB
exit 0
fi
if [ "$option" == "3" ]
then
OPTIONC
exit 0
fi
Here we have the if statements, quite straight forward really, what's happening is we are calling the 'option' variable, if this equals '1' for example then we call the OPTIONA method.
OPTIONA () {
echo "Hello world, this is option A"
}
As you can see option A echo's out the line "Hello world, this is option A". After each method the code 'exit 0' is run, this is just to exit the script cleanly.
Parameters
A good example of parameters would be if you wanted to write a script to remove all '.txt' files from a directory. In order to do this you could write a script which takes in the directory path as a parameter. This makes the script more flexible, it gives the user the choice of which directory input. The bash shell supports up to 9 parameters which are represented in the script as '$1' (if you remember awk this might make more sense) 1 being the number of the parameter.
Example
#!/bin/bash
(cd $1 && rm *.txt)
This is a very simple example, to run this script (lets call it remove.sh) you would do the following:
./remove.sh /home/zdrenka/pictures
The script would then assign ' /home/zdrenka/pictures' to '$1' and that's what we call a parameter, there are loads of useful ways in which to use this facility.
Since I started scripting I have picked up bits and bobs which can make your script look pretty cool, one of which is colours. Colours can make a script look that little bit more impressive. When I write a script one of the first things I do is paste this at the top:
################################################
# Define colours
################################################
txtund=$(tput sgr 0 1) # Underline
txtbld=$(tput bold) # Bold
txtred=$(tput setaf 1) # Red
txtgrn=$(tput setaf 2) # Green
txtylw=$(tput setaf 3) # Yellow
txtblu=$(tput setaf 4) # Blue
txtpur=$(tput setaf 5) # Purple
txtcyn=$(tput setaf 6) # Cyan
txtwht=$(tput setaf 7) # White
txtrst=$(tput sgr0) # Text reset
the '#' are obviously comments. I wrote this a while ago and now have it saved as a template, I use the colours to highlight errors, its very useful when you have a lot of text on the screen. to use these colours we just add them to the text as shown below:
echo "${txtbld}${txtred}ERROR File does not exist ${txtrst}";
The 'txtrst' variable will reset the text back to its original colour, you need this otherwise all text below your statement will be red.
Another useful command I have come across is the 'set -e' command. Put this at the top of your script and the script will automatically quit if it comes across and error, this is good practise as if you hit an error its not usually a good idea to carry on with the script. Although if you would like to have this command skip a certain part of the code you can surround that part with 'set +e' and 'set -e'. for example:
set +e
cp *.txt *.jar *.war /home/zdrenka/Java
set -e
Without these my script would stop once it hit this cp command as there may not be any '.war', '.txt' or '.jar' files in my Java directory which would technically throw up an error.
I hope this quenches your thirst for knowledge!
Thursday, 12 April 2012
SCP Command Examples
SCP stands for 'secure copy' its a good way of sending files over a network securely. To use this command you need to have SSH installed.
(for those of you who don't know SSH (secure shell) is a protocol for securely connecting to a remote machine). A good example of SCP is shown below:
scp somefile someUser@ipaddress
Its quite simple to use really, you'll find it useful if you use SSH a lot as I do.
Below I have listed a few fun/interesting scp commands. Enjoy :D
Edit a file on a remote host using VIM
vim scp://auser@ahost//path/to/somefile
I find this useful when you don't want to take the file, only edit it.
Compare a remote file with a local file
vimdiff afile.txt scp://[user@]host/<anotherfile.txt>
Again this is just as useful as the one above. you can also use this for coloured difference between to remote hosts.
vimdiff scp://user@server//path/to/file scp://user@server//path/to/file
SCP with Compression
scp -C 10.0.0.4:/tmp/backup.sql /path/to/backup.sql
-C - Compression
useful for all the DB heads out there.
SCP from one machine to another while logged into a third
scp user@hostb:file user@hostc:
scp a file from host b to host c while logged into host a. kinda fun I think
SCP a file to the same location on the remote machine
scp filename root@remote:`pwd`
this is very useful, I mean you could just write the path...but the pwd is there for a reason!
(for those of you who don't know SSH (secure shell) is a protocol for securely connecting to a remote machine). A good example of SCP is shown below:
scp somefile someUser@ipaddress
Its quite simple to use really, you'll find it useful if you use SSH a lot as I do.
Below I have listed a few fun/interesting scp commands. Enjoy :D
Edit a file on a remote host using VIM
vim scp://auser@ahost//path/to/somefile
I find this useful when you don't want to take the file, only edit it.
Compare a remote file with a local file
vimdiff afile.txt scp://[user@]host/<anotherfile.txt>
Again this is just as useful as the one above. you can also use this for coloured difference between to remote hosts.
vimdiff scp://user@server//path/to/file scp://user@server//path/to/file
SCP with Compression
scp -C 10.0.0.4:/tmp/backup.sql /path/to/backup.sql
-C - Compression
useful for all the DB heads out there.
SCP from one machine to another while logged into a third
scp user@hostb:file user@hostc:
scp a file from host b to host c while logged into host a. kinda fun I think
SCP a file to the same location on the remote machine
scp filename root@remote:`pwd`
this is very useful, I mean you could just write the path...but the pwd is there for a reason!
Sunday, 18 March 2012
Netcat Command Examples
The netcat command is used for an awful lot when it comes to TCP(Transfer Control Protocol) or UDP (User Datagram Protocol). It's mainly used to open TCP connections and send UDP packages to remote computers, although it can also be used for port scanning. In English the netcat command can be used to send data over a network.
Below I have listed a few basic commands to get you familiar with the program. Netcat can be abbreviated to nc in the shell.
Display Information sent by Browser
nc -l 8000
have netcat listen on port 8000, point browser to http://localhost:8000/ and see the information sent. Netcat will terminate once the browser has been closed.
-l = means that Netcat will listen to a connection rather then try and create one.
Sharing a file with netcat through port 80
nc -v -l 80 < file.txt
With this command you can share a file across a network, from a second machine, open a browser and go to http://ip-address-of-first-machine/ to access the file.
-v = Have nc give out a more verbose output(more detail).
-l = listen and do not create (as mentioned above).
80 = the HTTP port.
< file.txt = send the file over port 80.
Tail a log file over a network
tail -f error_log | nc -l 1234
This will serve a log file over a network through port 1234. This can be very useful if you have a big office.
to view this point a browser to the specified server/port (eg. 192.168.0.1:1234) and you can view the log file updates in real time! which is very cool.
tail is a command used to show the bottom of a file.
-f = this will allow the file to amended in real time.
Netcat as a Port-scanner
nc -v -n -z -w 1 AnIPaddress 22-1000
This should only be used on your own network with any permission required.
-v = Verbose (as explained above).
-n = Do not do a DNS or service look up on the specified address, host name or port.
-z = This is to specify that nc should just scan for listening daemons and not to send data to them (this means to look for open ports and not send anything to them).
-w 1 = this option sets a time-out period, if the connection is idle for more then 1 second then the connection is silently closed.
22-1000 = This will scan ports 22 to 1000 with the intent to find an available port.
Check the server is up, if not mail me
nc -zw2 www.example.com 80 || echo http service is down | mail -s 'http is down' admin@example.com
This is something I use whilst on 24 hour support. It's long but easy to understand.
-zw2 = This switch is a combination of -z and -w which are explained above. The switch will scan for listening daemons (without sending data to them) and have a time-out range of 2 seconds.
The command checks the server using port 80 (HTTP), if it received a time-out message it will send an email with the subject 'http is down' and 'http service is down' as the content to the specified address. Simple!
I hope this gives you a decent grounding for understanding Netcat, I find the best way to understand a program is to mess about with the commands and read through the man files. Any questions feel free to comment.
Below I have listed a few basic commands to get you familiar with the program. Netcat can be abbreviated to nc in the shell.
Display Information sent by Browser
nc -l 8000
have netcat listen on port 8000, point browser to http://localhost:8000/ and see the information sent. Netcat will terminate once the browser has been closed.
-l = means that Netcat will listen to a connection rather then try and create one.
Sharing a file with netcat through port 80
nc -v -l 80 < file.txt
With this command you can share a file across a network, from a second machine, open a browser and go to http://ip-address-of-first-machine/ to access the file.
-v = Have nc give out a more verbose output(more detail).
-l = listen and do not create (as mentioned above).
80 = the HTTP port.
< file.txt = send the file over port 80.
Tail a log file over a network
tail -f error_log | nc -l 1234
This will serve a log file over a network through port 1234. This can be very useful if you have a big office.
to view this point a browser to the specified server/port (eg. 192.168.0.1:1234) and you can view the log file updates in real time! which is very cool.
tail is a command used to show the bottom of a file.
-f = this will allow the file to amended in real time.
Netcat as a Port-scanner
nc -v -n -z -w 1 AnIPaddress 22-1000
This should only be used on your own network with any permission required.
-v = Verbose (as explained above).
-n = Do not do a DNS or service look up on the specified address, host name or port.
-z = This is to specify that nc should just scan for listening daemons and not to send data to them (this means to look for open ports and not send anything to them).
-w 1 = this option sets a time-out period, if the connection is idle for more then 1 second then the connection is silently closed.
22-1000 = This will scan ports 22 to 1000 with the intent to find an available port.
Check the server is up, if not mail me
nc -zw2 www.example.com 80 || echo http service is down | mail -s 'http is down' admin@example.com
This is something I use whilst on 24 hour support. It's long but easy to understand.
-zw2 = This switch is a combination of -z and -w which are explained above. The switch will scan for listening daemons (without sending data to them) and have a time-out range of 2 seconds.
The command checks the server using port 80 (HTTP), if it received a time-out message it will send an email with the subject 'http is down' and 'http service is down' as the content to the specified address. Simple!
I hope this gives you a decent grounding for understanding Netcat, I find the best way to understand a program is to mess about with the commands and read through the man files. Any questions feel free to comment.
Tuesday, 13 March 2012
Mplayer Command Examples
Mplayer is a media player, its mainly used to play audio or video.
What I like about it is how you can use it in the command line.
For example to play a audio file.
mpalyer mediafile.mp3
Its as simple as that.
For a video its slightly different. Mplayer does not support dvd menu's which is a bit of a pain really so you need to do the following
mplayer dvd://titlenumber
obviously title number needs to be replaced with your desired title number. Xine is a lot better for DVD menu's but that's a whole other kettle of fish
If you wish to move forward in a track or film there are a whole load of keyboard short cuts available here
I know your thinking "what's the big deal?" but the real reason I use mplayer is to rip audio from video files.
I use this a lot in conjunction with the youtube-dl program.
mplayer -ao pcm -vo null -vc dummy -dumpaudio -dumpfile someVideo nameOfAudioFile.mp3
I've taken the liberty off adding a selection of commands below:
Play all files in directory
$mplayer -playlist <(find "$PWD" -type f)
Stream Youtube directly to mplayer
$yt () mplayer -fs -quiet $(youtube-dl -g "$1")
use mplayer to play just the audio from a video
mplayer -vo null something.mpg
Shuffle songs in mplayer with no output
mplayer -shuffle $HOME/music/* $HOME/Dropbox/Music/*/* $HOME/Dropbox/Music/*/*/* etc. >> /dev/null 2>&1
What I like about it is how you can use it in the command line.
For example to play a audio file.
mpalyer mediafile.mp3
Its as simple as that.
For a video its slightly different. Mplayer does not support dvd menu's which is a bit of a pain really so you need to do the following
mplayer dvd://titlenumber
obviously title number needs to be replaced with your desired title number. Xine is a lot better for DVD menu's but that's a whole other kettle of fish
If you wish to move forward in a track or film there are a whole load of keyboard short cuts available here
I know your thinking "what's the big deal?" but the real reason I use mplayer is to rip audio from video files.
I use this a lot in conjunction with the youtube-dl program.
mplayer -ao pcm -vo null -vc dummy -dumpaudio -dumpfile someVideo nameOfAudioFile.mp3
I've taken the liberty off adding a selection of commands below:
Play all files in directory
$mplayer -playlist <(find "$PWD" -type f)
Stream Youtube directly to mplayer
$yt () mplayer -fs -quiet $(youtube-dl -g "$1")
use mplayer to play just the audio from a video
mplayer -vo null something.mpg
Shuffle songs in mplayer with no output
mplayer -shuffle $HOME/music/* $HOME/Dropbox/Music/*/* $HOME/Dropbox/Music/*/*/* etc. >> /dev/null 2>&1
Tuesday, 6 March 2012
Mail Command Examples
mail [-t] [-w] [ -m message_type ] recipient
The Mail Command is a good way (not the only way) of sending/receiving emails via the console.
Root is required for this:
mail someone@somewhere.com
This command will set up the email, it will then ask you for a subject, CC's and message.
once your done You need to type . (dot) to send the email. Then Enter.
It's better if you use the following command:
mail -s "subject" too@somewhere.com -- -r "from@somewhere.com" -f </path/to/a/file>
This command will send an email with a subject to an address from the address specified, the
-f </path/to/a/file> is the file which contains the information,in my case an error log.
this can be shorted to:
mail -s "subject" someguy@someplace.com < atextfile.txt
I often find myself using this command instead, its quicker to type...I should really make a script to do all this for me.
The Mail Command is a good way (not the only way) of sending/receiving emails via the console.
Root is required for this:
mail someone@somewhere.com
This command will set up the email, it will then ask you for a subject, CC's and message.
once your done You need to type . (dot) to send the email. Then Enter.
It's better if you use the following command:
mail -s "subject" too@somewhere.com -- -r "from@somewhere.com" -f </path/to/a/file>
This command will send an email with a subject to an address from the address specified, the
-f </path/to/a/file> is the file which contains the information,in my case an error log.
this can be shorted to:
mail -s "subject" someguy@someplace.com < atextfile.txt
I often find myself using this command instead, its quicker to type...I should really make a script to do all this for me.
My Top 5 SSH Commands
SSH(Secure Shell) is a network protocol for secure data communication, unlike Telnet SSH connects via a secure channel over an insecure network.
This means that data sent over SSH is encrypted.
Below I have added some useful commands for using SSH, Have Fun :)
I've had the problem where I need to connect to my PC at work, unfortunately
the IP address is unreachable, but it is reachable from the server at work.
Operation SSH man-in-the-middle is in effect!
ssh -t reachable_pc ssh unreachable_pc
This is good for copying a file to your work machine and having it sit
in your X11 buffer (clipboard) so then just right click and paste contents of the file
ssh user@host cat /path/to/a/text/file | xclip
This is very handy if you need to connect to a large number of hosts
it copies the SSH keys thus giving you password-less logins :D!
ssh-copy-id user@host
This is a strange one but kinda fun, I found this through Google so its not mine
this will output the sound from you microphone to a remote computers speakers via SSH
funky eh?!
dd if=/dev/dsp | ssh -c arcfour -C username@host dd of=/dev/dsp
One of the best things about SSH is Tunnelling, with the above command you can
view a remote hosts current website by going to http://localhost:2005
ssh -N -L2005:localhost:80 remotehost
This means that data sent over SSH is encrypted.
Below I have added some useful commands for using SSH, Have Fun :)
I've had the problem where I need to connect to my PC at work, unfortunately
the IP address is unreachable, but it is reachable from the server at work.
Operation SSH man-in-the-middle is in effect!
ssh -t reachable_pc ssh unreachable_pc
This is good for copying a file to your work machine and having it sit
in your X11 buffer (clipboard) so then just right click and paste contents of the file
ssh user@host cat /path/to/a/text/file | xclip
This is very handy if you need to connect to a large number of hosts
it copies the SSH keys thus giving you password-less logins :D!
ssh-copy-id user@host
This is a strange one but kinda fun, I found this through Google so its not mine
this will output the sound from you microphone to a remote computers speakers via SSH
funky eh?!
dd if=/dev/dsp | ssh -c arcfour -C username@host dd of=/dev/dsp
One of the best things about SSH is Tunnelling, with the above command you can
view a remote hosts current website by going to http://localhost:2005
ssh -N -L2005:localhost:80 remotehost
Sunday, 4 March 2012
Beginners Guide to Shell Scripting
It has come to my attention that a lot of people need a beginners' guide to shell scripting, therefore I am obliged to provide one.
The shell
The Terminal
The Console
Shell scripting in Linux isn't to difficult to master, although it helps if you have some programming background. We'll start by explaining the shell.
The shell
The shell is a program which actually processes commands and returns output. Bash is the most popular Linux shell as it contains standard features such as foreground and background processes and command history.
A question that is asked a lot is: "what's the difference between the Console and the Terminal?"
The Terminal
The Terminal is a wrapper program which runs a shell. This was once a physical device which consisted of not much more then a monitor and keyboard. As time moved on this became software and can be used to access the shell and run commands.
The Console
The Console is a special kind of Terminal, the Console used to be used for low level communication with the operating system. These days it is used in much of the same way as the Terminal.
In conclusion there isn't really much difference between the two, they both have separate pasts but are now essentially used for the same thing.
Writing Scripts
I recommend using Sublime to write your scripts as it has a lot of great tools to make use of, although if you are unfortunate enough to be using a Windows system I recommend Notepad++.
At the top of each script the we have a Shebang line. It consists of a number sign and an exclamation point character (#!), followed by the full path to the interpreter such as /bin/bash. All scripts under UNIX and Linux execute using the interpreter specified on a first line. Then we would have the command we wish to execute.
#!/bin/bash
echo Hello Linux!
The above script when run will display the words "Hello Linux" in the console/terminal. It uses the echo command which does what it says on the tin, it will echo out the words Hello Linux!.
This is one way of doing it, however the use of variables is more elegant. A variable stores data in memory and can be called at different times within the script, it can also be changed and amended throughout the script. Below is an example of Hello Linux with variables.
#!/bin/bash
VAR="Hello Linux!”
echo $VAR
Next we must learn how to use the chmod command, this is use to set the permissions for the script we have just created.
Chmod
the chmod command (change mode) is a command that lets the user control how much access it should permit to a file. I have a post with more details on this command here In order to run our script we must first use chmod to make it executable to do this I use the command
chmod +x helloLinux.sh
the +x makes the script executable for ALL users, if you just want it to be exe for you then u+x is for you.
After this command a simple ./helloLinux.sh will run the script.
Hopefully this should give you a good grounding for writing your own shell scripts. For a more advance guide try here
Any questions are welcome!
Thursday, 1 March 2012
Grep Command Examples
grep [options] PATTERN [FILE...]
The grep command basically searches though a file for lines which match the given pattern, for example:
grep apple fruit.txt
this will output all lines which contain the word apple, its as easy as that!
You could also specify a search in more then one file with the '*' wildcard
grep apple *.txt
like all good commands grep lets you use regular expressions such as the following:
grep ^a fruit.txt
the above line will return all lines which begin with the letter a.
Once you get the hang of grep you can do some pretty cool things, heres a command which I use a lot (provided by Sean, he know's who he is), it looks
complicated but I'll break it down:
grep --color=always -niC number "string" filename | less -R
--color=always - this colours the pattern to make it jump out at you when searching
-niC number
'n' - would be the line number, this isn't always needed so it can be removed.
'i' - is to match both upper and lower-case
'C number' - is the number of lines surrounding the matches you want to show, the alternative is A and B which is to show lines before or after the match
"string" - The pattern, but then you already knew that ;)
file name is the file you are searching, you can use the '*' wildcard as mentioned above or you can type another file name next to it which will also work
| less -R
this is the pipe that lets you scroll down the output of the search, the -R is the necessary command to let the colour work with the scrolling
basically this makes it easier to read.
you can also save the output as a .txt file by using | tee output.txt and appending it to the end of the grep command, the cool thing is that the colour stays in the output.txt!
There are loads of useful switches to use with grep try 'man grep' in the console
The grep command basically searches though a file for lines which match the given pattern, for example:
grep apple fruit.txt
this will output all lines which contain the word apple, its as easy as that!
You could also specify a search in more then one file with the '*' wildcard
grep apple *.txt
like all good commands grep lets you use regular expressions such as the following:
grep ^a fruit.txt
the above line will return all lines which begin with the letter a.
Once you get the hang of grep you can do some pretty cool things, heres a command which I use a lot (provided by Sean, he know's who he is), it looks
complicated but I'll break it down:
grep --color=always -niC number "string" filename | less -R
--color=always - this colours the pattern to make it jump out at you when searching
-niC number
'n' - would be the line number, this isn't always needed so it can be removed.
'i' - is to match both upper and lower-case
'C number' - is the number of lines surrounding the matches you want to show, the alternative is A and B which is to show lines before or after the match
"string" - The pattern, but then you already knew that ;)
file name is the file you are searching, you can use the '*' wildcard as mentioned above or you can type another file name next to it which will also work
| less -R
this is the pipe that lets you scroll down the output of the search, the -R is the necessary command to let the colour work with the scrolling
basically this makes it easier to read.
you can also save the output as a .txt file by using | tee output.txt and appending it to the end of the grep command, the cool thing is that the colour stays in the output.txt!
There are loads of useful switches to use with grep try 'man grep' in the console
Wednesday, 29 February 2012
Awk Command Examples
awk <pattern> '{print <stuff>}' <file>
Karl 50 20 90 30
William 43 89 12 56
Natalie 93 87 46 10
Awk lets you preform calculations such as the following:
awk '{ print "Average for ", $1,"is", ($2+$3+$4+$5)/4}' score.txt
Average for William is 158
Average for Natalie is 228.5
The awk command something you will almost certainly come across when bash scripting, amongst other things, its very useful for preforming calculations on numbers which are in a file. for example:
say you have a file called 'scores.txt' with the following information
say you have a file called 'scores.txt' with the following information
Karl 50 20 90 30
William 43 89 12 56
Natalie 93 87 46 10
Awk lets you preform calculations such as the following:
awk '{ print "Average for ", $1,"is", ($2+$3+$4+$5)/4}' score.txt
What's happening in the above command is awk is making use of the print function which basically does what it says. Each '$number' is a variable, for as shown above '$2' on the first line would equal 50, all these variables are being added together and then divided by the total number which gives us the average.
the output of this will be:
Average for Karl is 167.5the output of this will be:
Average for William is 158
Average for Natalie is 228.5
I find this very useful for scripts, you can find out more with 'man awk'.
Tuesday, 28 February 2012
Crontab Examples
Cron is kinda like the at command only a hell of a lot better, but essentially it is used to running commands/scripts in the background at regular intervals.
To create or edit a crontab file add the following command into your shell: crontab -e This will bring up a screen much like the following:
Your commands will go at the bottom of this page. Crontab is a simple text file with a list of commands meant to be run at a certain time. The commands are controlled by Cron which executes them in the background. If you want to set a script to run on you system at regular intervals something like the below command should be added to the crontab file shown above:
30 15 16 2 1 /scripts/test/runThisScript.sh
This doesn't seem to make much sense at first glance but let me try and explain. The first five numbers indicate at what intervals you wish to run your command/script, this is the breakdown
1 Minute 0-59
2 Hour 0-23 (0 = midnight)
3 Day 1-31
4 Month 1-12
5 Weekday 0-6 (0 = Sunday)
* = all
So we can see that I am asking Cron to run my script at 15:30 on the 16th February, Monday, plus every Monday in February. You could have it run everyday at 15:30 no matter what the date is:
30 15 * * * /scripts/test/runThisScript.sh
Make sure you give each a new line after each cron job, otherwise the job will not be executed...god knows why.. To save a cron-job press 'ctrl-K-X'. One of the jewels of Linux in my opinion.
more information can be found with 'man crontab' from your console.
To create or edit a crontab file add the following command into your shell: crontab -e This will bring up a screen much like the following:
Your commands will go at the bottom of this page. Crontab is a simple text file with a list of commands meant to be run at a certain time. The commands are controlled by Cron which executes them in the background. If you want to set a script to run on you system at regular intervals something like the below command should be added to the crontab file shown above:
30 15 16 2 1 /scripts/test/runThisScript.sh
This doesn't seem to make much sense at first glance but let me try and explain. The first five numbers indicate at what intervals you wish to run your command/script, this is the breakdown
1 Minute 0-59
2 Hour 0-23 (0 = midnight)
3 Day 1-31
4 Month 1-12
5 Weekday 0-6 (0 = Sunday)
* = all
So we can see that I am asking Cron to run my script at 15:30 on the 16th February, Monday, plus every Monday in February. You could have it run everyday at 15:30 no matter what the date is:
30 15 * * * /scripts/test/runThisScript.sh
Make sure you give each a new line after each cron job, otherwise the job will not be executed...god knows why.. To save a cron-job press 'ctrl-K-X'. One of the jewels of Linux in my opinion.
more information can be found with 'man crontab' from your console.
Subscribe to:
Posts (Atom)