Linux Automation and Scripting Guide
Linux and UNIX based operating systems have had all the basic ingredients for relatively painless automation built in for ages. You can make it do anything at any time, by any schedule or any trigger. Your wish is its command!
These basic ingredients are shell scripting in bash, a few key configuration files, and cron. Add to that various utilities that allow you to set up custom keyboard shortcuts or mouse triggers and you're ready to make it sing to your tune!
Let's first cover the most powerful part of this process. Scripting is just making the system follow a certain script the way an actor would follow a script of a movie. It may be similar to programming, but it's typically simpler, and can be looked at as just a fancy way of stringing multiple commands together in a desired order.
Those of you who already have experience using the command line on Linux have already met Bash, the "Bourne Again SHell", as that's what all those commands are typed into. The same commands you can use on the bash command line you can use in bash scripts.
This is best demonstrated by a relatively simple example, which also happens to be relevant in the context of automation. Here's a bash script that exports a list of installed packages, dumps a mysql database, and then syncs all of this along with a specified directory to the Amazon S3 cloud.
echo BEGINNING BACKUP RUN >> ~/backups/log/backuprun.log
# Go to backups dir.
# List installed packages.
dpkg --get-selections > installed_packages
echo Installed packages listed >> ~/backups/log/backuprun.log
# Dump mysql stuff
mysqldump -uusername database > ~/backups/mysql/database.sql
echo MySQL Databases dumped >> ~/backups/log/backuprun.log
# Sync to S3
echo S3 Synced >> ~/backups/log/backuprun.log
# Log the date and time
date >> ~/backups/log/backuprun.log
echo BACKUP RUN COMPLETE >> ~/backups/log/backuprun.log
echo =================== >> ~/backups/log/backuprun.log
What it does is the following, in sequence:
- Logs the beginning of the backup run to the backuprun.log file
- Changes current directory to ~/backups
- Lists all installed packages to the installed_packages file
- Exports the database to a database.sql file
- Executes ./backup-home, which is actually another bash script that syncs the whole home directory, including all of the exported files above, to Amazon S3 using the s3sync program
- Logs the date and the completion of the backup run and exits
This example script is a stripped down version of an actual script that has been running for years without trouble.
Once you understand that a bash script is just a sequence of commands you can see why it's a perfect tool for automating various tasks. Literally every command and program you've got on your Linux installation right now, with all of its available command line options, can be programmed into a script like this in any sensible sequence you like.
Another cool example could be a script that allows you to, in one double click, launch a custom set of programs that are a part of your typical work environment and workflow. For example if you are a graphics designer who works with both GIMP and Inkscape, and also have a working directory with all of your projects you could have a bash script that launches both of those programs and a file manager in that directory in just one click.
Or if you're a music producer on Linux using multiple programs like a sequencer, an audio editor, some synths, and so on all connected through a Jack Audio Connection kit you can have a script that launches all of these automatically in sequence. Even when your work environment is complex, a simple script can launch it in an instant.
Speaking of executing scripts, for them to work they need to be made into executable files rather than regular files. So if you've got a script called musicapps.sh you would run this command on it to make it executable:
chmod +x musicapps.sh
Or right-click on it, go to properties, permissions, and select to make it executable. This works similarly no matter which desktop environment or file manager you're using.
And that's a basic overview of bash scripting in the context of automation. There is truly a lot more you can do with bash scripts, and there are many great resources for learning more online, such as the Bash Guide to Beginners or the Advanced Bash-Scripting Guide.
Writing a script like above just gives you a yet another executable tool like all the other installed programs, but you still have to launch it yourself. If you want it to launch by itself at a predetermined time, and repeat within a given interval you can schedule it with cron.
Cron is a simple service that executes given scripts at preconfigured times and intervals, and these intervals are set up in the /etc/crontab configuration file.
On modern distributions like Ubuntu the default /etc/crontab will already be preconfigured for launching scripts in predetermined directories hourly, daily, weekly, and monthly. These directories are /etc/cron.hourly, /etc/cron.daily, /etc/cron.weekly, and (you guessed it) /etc/cron.monthly.
This means that setting a script or a command to run at one of these intervals automatically is as easy as putting it in one of those directories. If you have a backup.sh script (like the one above perhaps), and want it to run every week, you would simply place it into /etc/cron.weekly/. Of course, you'd need superuser privileges for this (sudo). If your terminal is in the same directory as your script you'd just copy it over with this command:
sudo cp backup.sh /etc/cron.weekly/
However, you might want to keep the script handy somewhere in your home directory in case you ever want to edit it without having to copy it over each time. Then instead of copying it into the cron.hourly directory you can just link it, like this:
sudo ln -s backup.sh /etc/cron.weekly/backup.sh
This would create a backups.sh link in cron.weekly to the real backup.sh in your home, therefore running your script every week.
That said, you might still want to set very specific times when you want to run your scripts, not just whenever it is preconfigured to run every hour, day, week, or month. Maybe you want it at exactly the right minute every day. You can configure this by editing the mentioned /etc/crontab file by adding a new line similar to the existing lines, but with your own customized time values.
This is how the /etc/crontab configuration file typically looks like:
# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.
# m h dom mon dow user command
17 * * * * root cd / && run-parts --report /etc/cron.hourly
25 6 * * * root cd / && run-parts --report /etc/cron.daily
47 6 * * 7 root cd / && run-parts --report /etc/cron.weekly
52 6 1 * * root cd / && run-parts --report /etc/cron.monthly
The syntax of it is simple. Numbers represent minutes (0-59), hours (0-23), days of month or "dom" (1-31), month or "mon" (1-12), and day of week or "dow" (0-6).
Asterisks (*) are wildcards, which basically just mean "every time". The "root" above is just the user which will run the command, and all the stuff after that is the actual command to run.
So the first line simply says that cron should run a given command every hour at the 17th minute, and do so every day of every week of every month. The second line says it will run every day at 6:25 AM, every week of every month. Third one will run at day 7 of every week and every month at 6:47 AM, and the fourth one will run first day of every month at 6:52.
The commands in the default crontab file above simply change the current directory to / (root directory), and (&&) execute the run-parts command with the --report option on specified directories (/etc/cron.hourly, and so on), which runs all of the scripts found in that directory.
Now if you want to run your backup.sh script, or any other script, at exactly midnight and 11 seconds every day you could simply add this line:
11 0 * * * root cd ~ && ./backup.sh
The ~ symbol represents whatever your home directory is, where for the sake of this example we assume your script is. The ./backup.sh launches the script from the current directory.
Of course, you could also create your own cron directory anywhere you want, like ~/cron.everynight, put your scripts there, and add a line like this to have cron run those scripts at exactly the times you want:
11 0 * * * root cd / && run-parts --report ~/cron.everynight
Needless to say, you can also adjust the default times of the lines already present in the file. Just change the time values.
Finally, for all of this to run as expected you'll need to have the cron service running on your system. If it isn't already, you can start it with this:
sudo /etc/init.d/cron start
After that it will start automatically at every boot, keep an eye on your schedules, and execute them at specified times. You can learn more by exploring the manuals available via the
man cron and
man crontab commands.
If all of this is a bit much and you're on desktop system you can also use a graphical program like GNOME Schedule. All it does is offer you an easier way of editing the cron configuration above. You can also use some of the alarm clock apps, like the Alarm Clock (gnome-automate or alarm-clock package), KAlarm, and KTimer! No joke; besides ringing, playing a song, or showing a message at a given time they're typically able to also run a given command.
Running at Startup
If you want scripts or programs to run automatically when you boot the system there are a couple of files you can edit, both in your home directory. One is .xinitrc, and the other is .bashrc. The difference is that .xinitrc runs every time you start X11, or the windowing system in Linux, whereas .bashrc will be read every time the shell is initiated, which is at every system boot.
This makes .xinitrc suitable if you want to automatically launch graphical programs and scripts or commands related to them. Otherwise it's safest to just use .bashrc.
That said, to make the script run on startup simply write in the line you would use on the terminal to launch the program or script, and save the file. If the .bashrc or .xinitrc file doesn't exist already, create it. Note that it does have to be in your home directory (e.g. /home/daniel/).
The line you would add to run the example backup.sh script from above would then look something like this:
Literally nothing more would be needed. You can add multiple commands and scripts like this each in their own line.
You can also set startup programs graphically if you're on any of the more complete desktop environments like GNOME, Unity, KDE, Xfce, and so on. There's typically a settings dialog dedicated to this purpose. Otherwise you can use programs like the BootUp-Manager.
Custom keyboard shortcuts
Finally, you can bind launching commands, programs, and your scripts to a custom keyboard shortcut.
It's likely that your desktop environment already has this customizable through its settings panel where all you'd have to do is specify the command you want to run, whether it is launching your script, or a given program, and then specify your keyboard shortcut. Here's where to find these settings for popular environments.
GNOME 2 / MATE / Cinnamon / Unity (Ubuntu default)
Method 1: Go to System > Preferences > Keyboard Shortcuts and alter the settings there.
Method 2: Install and launch gconf-editor, and in it navigate to apps > metacity > global_keybindings where you can edit and add keyboard bindings to commands.
Method 3: Install and launch dconf-editor, navigate to org > gnome > desktop > wm > keybindings, and edit or add new bindings there.
In the Settings panel go to "Keyboard", and then "Custom Shortcuts" where you can add new ones. You can also find keyboard settings by searching for "keyboard" in the GNOME Shell.
Go to Settings, find Keyboard, and switch to the Application Shortcuts tab to edit and add shortcuts.
Go to System Settings > Shortcuts and Gestures and edit or add new shortcuts there.
While scripting and launching commands and scripts on schedule, boot or key presses give you tremendous power in automating certain processes there are some things where specialized automation applications are probably the best tool for the job. So to make the Linux automation story complete here's a brief overview of a few of them.
AutoKey allows you to add phrases and set them to be initiated using a hotkey or a keyboard abbreviation. This means you can set up a phrase that you might be using often, like perhaps "Thank you and Best Regards", set an abbreviation of "tybr", and so long as AutoKey is running you only need to say "tybr" and it will paste in the whole phrase.
Since phrases are essentially just shortcuts to longer strings of keyboard presses they also support recording keyboard and mouse actions, and then setting hotkeys and abbreviations for those as well, allowing you to automate some repetitive tasks you might be doing.
AutoKey also supports python scripting so you can write some powerful scripts to automate tasks, right within the AutoKey window, and run them with hotkeys and abbreviations. Worth noting is that there's a Window Filter option to disable a given automation within a given window, and also options to show a prompt and/or a notification whenever an automation is executed. How well will all of this work? Your mileage may vary, but it is worth trying out. You can check the AutoKey documentation here.
Actionaz is basically a full blown graphical scripting tool. Instead of typing up the script you construct it by dragging and dropping various actions from the left hand sidebar until you end up with a complete action list, or a script.
This could make it a fair bit easier to automate complex tasks, but you do need to know well what you want to accomplish and have a basic understanding of what each of the actions does and how it works. Suffice it to say that some learning curve is present.
That said this could be a powerful and expedient way to automate many tasks.
This is basically an auto clicker app for Linux, and it does exactly what it says. It allows you to script mouse clicks and then initiate them with a custom keyboard shortcut. It doesn't to live recording so you have to script each click manually. This can still help automate mouse click driven tasks. The app is sold for $10 in the Ubuntu Software Center, but you can download it from the web site and try it out for free.
Robotux is a very straightforward key