Episode 032 – cat

The cat command concatenates files or standard input to standard out by default. What this means is that the cat command will display the contents of a file to standard out so you can read it. The cat command is also very useful with redirection allowing you to combine multiple files together as, for example, with the split command. Recall that the split command divided a file into multiple smaller files. To restore the original file the cat command was used to cancatenate the smaller files back into one large file.

The usage of the cat command is quite simple. For instance, if we had a file called cat_test.txt with the following contents:

here is a cat test
I am adding a tab on the next line
there is a tab here
throw some blank lines in here
See how that goes

Issuing:

cat cat_test.txt

Will output the contents of the file to standard out:

here is a cat test
I am adding a tab on the next line
there is a tab here
throw some blank lines in here
See how that goes

There are a few switches that allow you to control the output of the contents. The -n, or –number, will number each line of output:

cat -n cat_test.txt

Produces:

cat screen shot with -n option

You can just number only the non-blank lines using the -b, or –number-nonblank, switch:

cat -b cat_test.txt

Produces:

cat screen shot with -b option

The -s, or –squeeze-blank, switch will remove repeated empty lines of output. This switch will not remove all blank lines, but only the second or more blank lines in a row. So in the example file the blank line after the fourth line is retained; but lines 6, 7, and 8 are “squeezed” out:

cat -sn cat_test.txt

Produces:

cat screen show with -sn options

To see the non-printing characters that are in a file use the -v, –show-nonprinting switch. The example file has been augmented with some dos non-printing characters. Catting the file now:

cat cat_test.txt

Produces

cat screen show no options

But:

cat -vn cat_test.txt

Shows the non-printing characters:

cat screen shot with -vn options

Note the ^M and ^V characters on lines 10 and 11 respectively.

The -E, or –show-ends, displays a “$” at the end of each line:

cat -E cat_test.txt

Produces:

cat screen show with -E option

To show tabs in a file use the -T, or –show-tabs, switch and tabs are replaced with “^I”:

cat -T cat_test.txt

Produces:

cat screen shot with -T option

There are a few aggregate switches that combine options:

  • -e combines -vE ( –show-nonprinting and –show-ends)
  • -t combines -vT ( –show-nonprinting and –show-tabs)
  • -a combines -vET ( –show-nonprinting, –show-ends, and –show-tabs)

If no file is specified with the cat command standard input is used. Similarly, using a “-” will signify to use standard input instead of a file:

cat

Issued like this by itself the cat command will echo back what is entered through standard input once the “enter” or “return” key is pressed.

The cat command has no pager functionality so all content that the command displays will scroll off the screen. Using a pipe, “|”, output from the cat command can be passed to a pager like less or more:

cat httpd.log | less

The cat command is a handy utility for viewing the contents of a file or redirecting the contents of one file into another. One example, culled from the Linux from Scratch documentation is to use cat as a quick and dirty text editor:

cat > test.sh << “EOF”

This example cats all standard in to test.sh until a single line containing “EOF” is listed. At which point cat closes the file and exits. Using cat like this or with other commands like cut, head, and tail one can perform some pretty sophisticated text manipulation.

Bibliography

  • man cat
  • info cat

If the video is not clear enough view it off the YouTube website and select size 2 or full screen.  Or download the video in Ogg Theora format:

Thank you very much!

Posted in Uncategorized | 5 Comments

Episode 031 – who

The who command allows you to see who is currently logged on to the Linux system you run the command on. If you are running a desktop system the who command may not seem that important. But if you are managing a server where multiple users could be logged in at a single time, the who command will alert you to who is logged into the system if you need to do some work that may impact their session. Issuing who by itself produces output similar to this:

who output

default output of who

Four columns are displayed here:

  • The username of the user logged in.
  • The terminal the user is logged in on.
  • When the user logged in.
  • The remote host name or X display the user logged in from/to.

In the example above there is only one users logged in and that user is showing as logged in 10 times. Note the second column, the terminal the user is logged in to. The first line shows tty1 which is a virtual terminal or virtual console. Way back in the early days of computing a keyboard and display corresponding to one instance of a terminal to give you access to the computer. A virtual terminal allows for more than one terminal connection to be used by a keyboard and monitor. The device files for a virtual terminal can be found in the /dev directory. In this case, /dev/tty1 the terminal dann is logged into. This is also the terminal that X is running from.

The remainging devices labeled pts/# are psuedo terminals. Like virtual terminals, psuedo terminals provide terminal-like access. In this case, the psudeo terminals are all running under an X-session, specifically X display :0.0. These are all Konsole sessions. But they could very well be ssh sessions:

who showing remote logins

who output showing remote logins

In this screenshot there are 2 different user accounts logged on: root and dann. The root account is logged in locally but the two dann accounts are logged in from different hosts.

By default the who command gets the information about who is logged in from the /var/run/utmp, although utmp could be in a different location. The utmp file keeps track of users currently logged into the system along with other system related information since boot. You can point who to a different file to look at by specifying the path to the file after who:

who /var/log/wtmp

In this example who will print out the history of everyone who logged into the system. The /var/log/wtmp file stores historical login information.

Aside from options and file there is one more arguement that who will take and that is “am i”:

who am i

This will print out the username of the user who issues this command along with the terminal the user is running on, the time logged in and the host or X display logged in from. Essentially this is just the who information for the current user and currently user only. There is also a flag that will report this information:

who -m

To help you better recall what each column means, the -H, or –heading, switch will display a heading row at the start of the output defining each column:

who output with headers

who output with headers

The -l, or –login, switch will show all terminals that are waiting for a login. Most Linux systems define 6 virtual terminals be available at boot: tty1-6. Those virtual terminals that have not had anyone log in on yet will be displayed by this switch along and the user name will show LOGIN.

The -u switch will add an addtional column to the output after the time logged in showing how long the user has been idle. The value here is in the format HH:MM. The range is 24 hours so any value over 24 hours will report as “old” and a “.” signfies activity in the last minute.

The -w, -T, –mesg, –message, or –writeable, switches report the status of users ability to accept messages. One of three values will be reported in the column after the user’s name:

  • + = The user is accepting messages.
  • - = The user is not accepting messages.
  • ? = The terminal device was not found.

The ability to accept messages is set with the mesg command. Most system shave ths turned off by default so to turn it on the user must issues:

mesg y

Now messages from non-root users can be sent to the user using the write command.

The –lookup switch will attempt to canonicalize hostnames via a DNS lookup.

The -q, or –count, switch only reports the usernames of the users logged in and the total users logged in. No other standard information is reported.

The who command can display information other than who is logged in. The following list of switches can be passed to who:

  • -b, –boot = Prints the date and time of the last system boot.
  • -d, –dead = This switch prints out a list of dead processes (these are processes that have been spawned by init that have since died, not a list of all processes that have died (i.e.; logging out of a virtual terminal or close a psuedo terminal)).
  • -p, –process = This lists the active processes that were spawned by the init command.
  • -r, –runlevel = This will report the current run-level of the init process.
  • -t, –time = Prints the last system clock change.

Some of these values will not be reported on systems that have moved from init to Systemd or Upstart, so your mileage may vary on what is returned.
The final switch to report is -a, or –all, which is a combination of:

-b -d –login -p -r -t -T -u

This tells who to display the last system boot, list dead process, show any terminals waiting for LOGIN, list any active processes spawned by init, list the last system clock change, print the message status, and report the idle time for each user in addition to the regular who output.

The who command is a useful utility in a multi-user environment that will help you get a grasp on the users logged into your system. Beyond that it can display some other important information regarding the system status since last boot. Some of who’s functionality may be limited by migration from init to systemd or upstart, so be aware of these limitations. Regardless, who remains an important utlity for your toolbox.

Bibliography:

If the video is not clear enough view it off the YouTube website and select size 2 or full screen.  Or download the video in Ogg Theora format:

Thank you very much!

Posted in Uncategorized | Leave a comment

Episode 030 – vmstat

The vmstat program is a tool for displaying information about virtual memory usage and is part of the procps suite of utilities. The procps package provides tools for browsing the /proc file system. The /proc filesystem is “psuedo” file system that is created and stored in memory. The /proc filesystem provides an interface to the running kernel data structures. Most files in /proc are readable by any user account. Most files are also only readable but there are a few files that the root account can write to. According to the man page vmstat makes use of the following /proc files:

  • /proc/meminfo – This file holds information about memory usage on the system. Aside from being used by vmstat, meminfo is also used by the free program. For a complete listing of what each line in this file mean see man 5 proc.
  • /proc/stat – This file list kernel and system statistics. The contents of this file depend upon what version of kernel you are using. For a listing of what entries may be in here and what they indicate see man 5 proc.
  • /proc/*/stat – /proc/[pid]/stat shows status information about the specified running process. See man 5 proc for a list of what may be in this file.
  • /proc/slabinfo – This file contains information about kernel caches. This file is only readable by the root user and while you can view the information in vmstat, you must run vmstat with elevated priviliges should you elect to report the slab information. See man 5 slabinfo for more information.

Virtual memory is a memory management technique that, to the system, provides more memory than is actually available. Instead of a one-to-one addressing of processes to physical memory, the Operating System maintains a virtual table space where “pages” of virtual memory are allocated to processes. These pages are mapped to a physical memory via a page table. This physical memory can be RAM or more slower allocated disk storage like a swap partition or swap file.  To best maximize system memory, the Kernel attempts to allocate physical RAM to active processes while moving inactive process to virtual memory. That is, an inactive program may have its page moved to the swap partition. Moving data from RAM to virtual memory can be done by either paging or swapping. Swapping is when an entire process is transferred from physical memory to virtual memory. Paging is when portions of a process are written to virtual memory. A page-out is the writing of the “page” to virtual memory and a page-in is reading the “page” from virtual memory. Paging is normal behavior for a system, but excessive paging can cause problems, especially excessive page-outs. If the tasks assigned to your system exhaust the physical RAM for all active processes, the Kernel may need to push some active process pages out to virtual memory slowing the performance of the system significantly. Using a tool like vmstat can provide valuable insights into the health of your system and help ferret out performance problems.

Executing vmstat by itself prints out one a summary of information based upon the last reboot of the system. The information is presented in a table format:

vmstat default output

vmstat default output

The first line details the category of information detailed in the columns below:

  • procs – Process information – information about processes running on the system:
    • r = The number of runnable process that are running or waiting for run time. A “runnable” process is a process that is waiting for the CPU to run. The schedule determines what runnable process to send to the CPU.
    • b = The number or processes in uninterruptable sleep. A process that is in uninterruptable sleep is waiting on hardware conditions to be able to complete. While in this state they cannot be interrupted.
  • memory – memory information – information about physical and virtual memory:
    • swpd = The amount of virtual memory used, that is block or disk storage that is used for memory
    • free = This is the amount of idle memory in the system. Idle memory is memory that is no longer used by a process by still has information in it. Idle memory has not yet been reclaimed.
    • buff = The amount of memory being used as buffers. Buffers are raw disk blocks that store information from a process not related to file data. Information like permissions, file location, etc. are stored here.
    • cache = The amount of memory being used for cache. Cache stores file data information for a process.
  • swap – swap informtaion – information about the swap file/parition usage
    • si = The amount of memory (pages) swapped in from disk.
    • so = The amount of memory (pages) swapped out to disk
  • io – input/output information – information pertaining to data transfer between block devices
    • bi = Blocks in received from a block device (disk)
    • bo = Blocks written out to a block device (disk)
  • system – system informtaion – basic overall system information:
    • in = Number of interrupts per second, including the clock interrupt. An interrupt is a signal by a hardware device or process that demands the CPU stop whatever it is currently processing and switch attention to the device or process raising the interrupt.
    • cs = Number of context switches per second. A context switch when the CPU stops and stores the state of a process to be restored at a later time or when a stored process is returned to active state to continue executing the process.
  • cpu – CPU information – information on process utilization of the CPU
    • us = This represents time spent running non-kernel code. This is the time the CPU spends running applications started by a user or called by an application started by a user. This is non-system call related processing.
    • sy = This represents system time. System time is time the CPU spends running system calls, calls related to the time the kernel is utilizing the CPU.
    • id = This represents idle time. Idle time is time the CPU is not currently processing a request. This does not mean there are no processes running, just that no process is requiring the CPU. This counter does not include i/o wait.
    • wa = This represents CPU Wait time where the CPU is waiting for I/O to complete.

This is the default reporting for vmstat. What gets reported and how can be altered with switches, which will be covered shortly. As stated calling vmstat by itself prints out a summary of system information from the last reboot. Two values can be passed to vmstat. The first it a delay in seconds and the second is a count, the number of cycles vmstat will run for. If you only specify the delay value vmstat, that is no count, vmstat will continue to report indefinetly. Specifying delay and/or cycles changes the reporting values from the last reboot to report the values reflecting the time between the last cycle. The value for delay and cycle must be a postive integer.

vmstat 2 5

This will cause vmstat to report statistics every 2 seconds for 5 intervals. The first report will reflect statistics from the last reboot. Each subsequent report will contain the statitics on the previous 2 seconds, the duration between cycles.

The -S, or –unit, switch changes the default output values from KB (1024 bytes) to one of the following:

  • k 1,000 bytes
  • K 1,024 bytes
  • m 1,000,000 bytes
  • M 1,048,576 bytes

Changing the unit will not effect the output of si, so, bi, and bo.

While vmstat is running it is smart enough to re-print the header if the values increment enough to cause the header to scroll off the screen. This behavior can be suppressed with the -n, –one-header, switch.

The -a, or –active, switch will show active and inactive memory information. This will introduce two new fields that replace the Memory Buffer and Cache statistics:

vmstat -a output

vmstat -a output

Memory

  • inact = Reports inactive page. An inactive page is a page that is no longer in use. Inactive pages can be in one of three states:
    • Dirty – This is a page that is no longer in use but has data that has been changed and needs to be written to disk.
    • Laundered – This is an interim state for a dirty page where it is having the contents moved from memory on to disk.
    • Clean – This is a state where either an inactive page does not have changed data or a dirty page has completed the laundered phase. Clean inactive pages can be deallocated.
  • active = The amount of pages in memory that are currently in use.

The -s, or –stats switch will output a table of event counts and statistics:

vmstat summary report

vmstat summary report

This display cannot repeat so neither a delay or cycle value will be accepted. The information reported is essentially the same information gathered from the default output just in a more user readable format. Note that some of the statistics are broken out in more detail in this format whereas they are added together in the default reporting. For instance, the default output combines both nice and non-nice user cpu time into just user time whereas the –stat view breaks these values out.

The -d, or –disk, reports disk statistics, replacing the default memory, system and cpu statistics:

vmstat -d output

vmstat -d output

The report details the following:

  • Reads
    • total = This is the total number of successful reads completed from the disk.
    • merged = This is the total number of grouped reads that resulted in one coplete I/O.
    • sectors = Sectors is the total number of sectors that were read successfully.
    • ms = The total number of milliseconds spent reading.
  • writes
    • total = Total number of writes completed successfully.
    • merged = Total number of grouped writes completed successfully resulting in one I/O.
    • sectors = Total number of sectors written successfully.
    • ms = Total number of milliseconds spent writing to the disk.
  • IO
    • cur = Reports any I/O currently in progress.
    • s = Reports any seconds spent for I/O.

Disk information is reported on a per block device basis. Partition information can be listed with the -p, –partition, switch and specifying the device:

vmstat -p /dev/sda1

vmstat partition report

vmstat partition report

Again, the default output is replaced with a report on the partiton listed:

  • reads = Total number of reads completed successfully from this partition.
  • read sectors = Total number of sectors successfully read from this partition.
  • writes = Total number of successful writes to this partition.
  • requested writes = Total number of write requests made to this partition.

Like the standard report, the disk report has a summary view reported using the -D, or –disk-sum, switch:

vmstat -D

vmstat disk summary report

vmstat disk summary report

The report is a summary of the information for all disks connected to the system. Like the –stat view, the –disk-sum report will not accept the delay and/or cycle value.

The -f, or –forks, switch will display the total number of forks since the last reboot. This values includes to sum of the following totals:

  • Forks = A fork call that duplicates a current process. This child process gets a new PID and has the it’s parent PID set to the PID of the process it forked from.
  • Vfork = Similar to a fork, when a process creates a Vfork, the process the Vfork was created from is temporarily suspended until the child process exits.
  • Clone = A clone is similar to a fork but a clone allos the new process to share part of the execution context with the calling process.

The –forks switch does not accept delay or cycle values.

Vmstat has the ability to report on slabs using the -m, or –slabs, switch. A slab is a memory managment mechanism for allocating kernel objects and is used by the kernel in controlling cache. This switch can only be used by an account with elevated privileges:

vmstat -m

vmstat slabinfo report

vmstat slabinfo report

The output is a less verbose summary of the /proc/slabinfo file:

  • Cache = Name of cache being reported.
  • num = The number of currently active objects in this cache.
  • total = The total number of available objects in this cache.
  • size = The size of each object in the cache.
  • pages = The total number of pages with at least one active object associated with this cache.

The –slabs switch does not accept delay or cycle values.

The vmstat tool is a handy utility for reporting on your systems virtual memory use and will give you some statistics on disk storage. This is a handy tool for helping uncover performance issues on a Linux system.

Bibliography


If the video is not clear enough view it off the YouTube website and select size 2 or full screen.  Or download the video in Ogg Theora format:

Thank you very much!

Posted in Uncategorized | Leave a comment

Episode 029 – ab – Apache BenchMark

The Apache BenchMark tool is a handy application to have in your toolbox that can perform a number of helpful tasks. First, the primary use for the tool is to simulate how your server may react under maximum load. Because of this, if you are experiencing intermittent issues with your HTTP server you may be able to force failures or errors by increasing the load against your server using Apache Bench Marking tool. Be aware, though, that this tool primarily reports requests per second of a single page and is not necessarily indicative of how a typical user may interact with your site.

The Apache BenchMark tool is included with the Apache software but may be available in your distrobution under the Apache-Utils package. You do not need to have the Apache Web Server running on the machine you want to run AB against and you can run it against any HTTP server, not just Apache. Only run this tool against a web server that you have permission to test. Running this tool against any website you desire can get you into trouble. Finally, you should not run AB on the same system that you are load testing against.

The Apache BenchMark tool’s executable is ab. The only parameter required to be passed to ab is the webpage you want to bench mark:

ab http://www.thelinuxlink.net/index.php

This will execute AB against the web server hosting the domain www.thelinuxlink.net and target the page index.php. This test is not very stressful as it will only perform one request against the server. You can specify the number of request you want to make with the -n switch:

ab -n 500 http://www.thelinuxlink.net/index.php

This will send 500 requests to the web server one at a time. Again, not very stressful against the server you want to benchmark. To really begin to stress the system include the -c, option which defines the number of concurrent connections per request:

ab -n 500 -c 100 http://www.thelinuxlink.net/index.php

This will again perform 500 requests to the index.php but instead of one request at a time will process up to 100 requests concurrently. This does not mean that it will process 100 concurrent request 500 times. To achieve that test you would have to run this command:

ab -n 50000 -c 100 http://www.thelinuxlink.net/index.php

Before continuing on to some of the other switches ab will take we should probably analyze the output of the command:

Apache BenchMark Output

Apache BenchMark output example

Here is a break down of the sections:

Apache BenchMark Server Summary

Apache BenchMark Server Summary

As ab runs it will indicate progress after each 10% or 100 (minimum value to trigger) requests of the total requests are made. Once the total number of requests have completed the summary results are displayed. The first five lines display some information about the environment being tested:

  • Server Software – The name or the webserver software and the version if reported.
  • Server Hostname – The hostname or domain name of the server benchmarking.
  • Server Port – The port the benchmark connected to the server with
  • Document Path – The path to the document you are connecting to.
  • Document Length – The size of the document
Apache BenchMark Test Summary

Apache BenchMark Test Summary

The section of information details the test that was run:

  • Concurrency Level – The number of concurrent requests made at one time.
  • Time Take for Tests – The overall time it took for the whole benchmark to complete.
  • Complete requests – Total number of successful responses to the requests.
  • Failed requests – Total number of unsuccessful responses to the requests. If this number is greater than zero further information will be presented showing the number of requests that failed due to connecting, reading, incorrect content length, or exceptions.
  • Write errors – This shows the number of errors that were made during the write process.
  • Total transferred – The number of bytes received from the server.
  • HTML transferred – The number of bytes received from the server that were from the document.
  • Request per second – The mean number of requests the server processed per second. The value is determined by the total number of requests divided by the number of seconds it took to complete the requests.
  • Time per request – The mean across all concurrent requests on how long it took per request.
  • Transfer rate – This is the rate of transfer in kilobytes recived from the server per second.
Apache BenchMark Connection Information

Apache BenchMark Connection Information

The third section details the overall connection time results presenting the minimum values, mean value, standard deviation, median, and maximum value:

  • Connect – The amount of time it took to connect to the web server and return the first bits of a response.
  • Processing – The amount of time it took the server to process a request.
  • Waiting – How long it took for the first byte of data to be returned from a request.
  • Total Time – How long it took to complete a request.

These values are in milliseconds. Again, note that there are 5 columns here:

  • Minimum – The shortest amount of time taken.
  • Mean – The average number of seconds per request.
  • +/- sd – The standard deviation in milliseconds for the mean.
  • Median – Median is the middle value of all requests. To get this value you would have to list all the result value in numerical order and find the middle number.
  • Max – The maximum number of milliseconds a request took.
Apache BenchMark Percentages

Apache BenchMark Percentages

The final bits of data report the “Percentage of requests served within a certain time.” These values are in milliseconds. In this example 50% of the requests completed in at the most 990 milliseconds. 90% of the requests completed in at the most 2320 milliseconds (about 2.3 seconds). All requests completed within 2654 milliseconds (2.6 seconds) which was the longest request time noted by the test.

Now that you have a basic understanding of how to interpret the results of the ab tool, let us return to covering some more of the switches.

The -k switch will turn on the KeepAlive feature. KeepAlive will keep the connection open between the webserver and the client to perform multiple requests within the same http session. The default for ab is to have this feature off and require a new connection for each request in the same session from the same client.

If you need to authenticate to the webserver to request the resource you can use the -A with like this:

-A username:password

The credentials are sent in base64 encoding. So be careful if you are not connecting to an secured channel.

You can run tests against https and if you need to specify the SSL/TLS protocol use the -f switch:

-f SSL2 | SSL3 | TLS1 | ALL

If you need to run your test through a proxy use the -X to define the proxy address and port:

-X proxy_ip:proxy_port

By default ab uses a GET request which returns the body of the document requested. You can switch to a HEAD request with the -i switch. A HEAD request will not return the body of the message, only the document headers, the meta-information.

You can sent POST and PUT information using the -p and -u switches respectively. Both of these switches require that a file with the POST or PUT contents be passed to the switch. Additionally, you must specify the content-type with the -T switch. The default content type is text/plain. For POST/PUT data you will probably want to set this to:

-T application/x-www-form-urlencoded

An example POST or PUT file might contain the following information:

name=dann&amp;email=dann@thelinuxlink.net&amp;country=us

You can change the size of the TCP send and receive buffers with the -b switch and specifying a size in bytes:
You can run ab for a period of time as opposed to a number of request using the -t switch and specifying a number of seconds:

ab -t 10 http://www.thelinuxlink.net/index.php

This will run the benchmark for 10 seconds with the default concurrency of 1 request.

There are a few switches to control the output of the ab results:

  • -d : Suppresses the percentage served information, the last block of data on the output.
  • -q : Suppresses the progress output while ab is running.
  • -S : Suppresses the median and standard deviation columns in the connection time results. This will also suppress the warning/error messages when teh average and median are more than one or two times the standard deviation apart.
  • -v : The -v switch takes a number from 1 (default) to 4 for verbosity of debug information:
    • 2 – Prints warnings
    • 3 – Prints response codes
    • 4 – Prints header information

The results of ab can be output in HTML table format with the -w switch:

ab -n 1000 -c 100 -w http://www.thelinuxlink.net/index.php

The output will be HTML code with the results after the percentage update in an HTML table. You can specify html formatting with the following switches:

  • -x : table attributes: -x “width=200px”
  • -y : tr attributes: -y “style=’text-align: center;’”
  • -z : td attributes: -z “style=’color: red;’”

You will probably want to redirect the output to a file otherwise the code will just go to standard out:

ab -n 1000 -c 100 -w http://www.thelinuxlink.net/index.php > abresults.html

The ab command will generate a comma separated value file with the -e switch:

ab -e ab_results.csv -n 1000 -c 100 http://www.thelinuxlink.net/index.php

This will still display the progress on standard out. The resulting CSV file contains two values: A percentage from 0 to 99 and the time it took to serve that percentage or requests in milliseconds.

The -g option will output the results in a tab separated format that can be imported in applications like gnuplot, IDL, Mathematica, or a spread-sheet application like Libre-SpreadSheet:

ab -e ab_results.tsv -n 1000 -c 100 http://www.thelinuxlink.net/index.php

The Apache Benchmark tool is a handy application to have in your tool kit to not only perform cursory load tests on a web server but to stress the server to help identify intermittent problems. For more information check out these links:


If the video is not clear enough view it off the YouTube website and select size 2 or full screen.  Or download the video in Ogg Theora format:

Thank you very much!

Posted in Uncategorized | Leave a comment

Episode 028 – Extended Attributes – lsattr and chattr

Back in episode 7 the chown command and UNIX file permissions were discussed. This episode is going to extend that and talk about file system attributes, or more appropriately named extended attributes. Recall that Linux file systems support three permission attributes: Read, write and execute for three different levels: Owner, owning group, and everyone else. Extended attributes, abbreviated xattr, add some more permissions or restrictions to the original three attributes. Extended attributes are currently supported in the ext file systems along with JFS, ReiserFS, XFS, Btrfs, and OCFS2 version 1.6. The extended attributes provide the following manipulable behaviors ordered by their labels:

  • a – append only: Writing to a file will only allow the file to be opened in append mode for writing. That is you cannot redirect output to overwrite the file, only append to it. Most normal file edit operations, like opening the file with a text editor, will most likely fail as the program will attempt to overwrite the file with the changes and “permission denied” will be displayed. This attribute can only be set by an account with superuser privileges.
  • A – Do not update Atime – access time. When the file is accessed do not update the access time (atime) attribute. This attribute can be a little tricky because if you want to see it in action you need to be aware of a few things. The behavior of atime is determine by how the file system is mounted. With noatime set the access time of the file is not updated when the file is accessed. With atime set use the kernel defaults for how atime is set. Now it could be set in strict mode where atime is always updated when the file is accesses or with relatime which only update the access time should it be older than the modification time when the file is accessed. So if you are monitoring the atime with stat on a file and you are accessing the file and not seeing this time stamp change, chances are the file system you are on is mounted with noatime or relatime set. You would actually have to modify the file and then access it (e.g; cat the file) before the atime would change.
  • c – compressed attribute: Compress sets the file to be compressed when written to on the disk. Data written to the file is compressed by the kernel before the file is written and when read, the file is uncompressed by the kernel for the read. Note that setting this attribute can incur overhead as the file will need to be compressed and uncompressed when written to or accessed. Note that this attribute is not honored by ext2 and ext3 file systems.
  • C – no copy-on-write: The standard behavior on most Linux file systems is that when a file is opened by multiple tasks at one time, instead of creating multiple copies of the information for each task, a pointer to the shared resource is used. Should a task write to the file, then a private copy is broken off for that task instead of manipulating the shared resource. This behavior can be turned off by setting the C attribute so that a separate, private copy is never created.
  • d -no dump: This marks the file as not being a candidate for backups when the dump command is used.
  • D – Sychronous Directory Updates: When a change is made to the file changes are written synchronously to the disk. What this means is that any changes to a file are immediately written to disk and available. This is particularly useful in a networked situation where multiple systems may be accessing a share. When a file is created in that share the directory is immediately updated so that it is visible to all systems accessing the share.
  • e – extents: This attribute indicates that the file is using extents for mapping the blocks on disk. Extents replaced traditional block mapping schemes (found in ext2 and ext3) and define a contiguous range of physical blocks for storage of the data. This attribute is set by the file system and cannot be altered by chattr.
  • E – This is an experimental attribute that may be set by compression programs to indicate that a compressed file has a compression error. This attribute cannot be set or changed with chattr.
  • h – This is not an attribute you can set. Its presence indicates that the file is storing its blocks in units of the file system block size as opposed to the units of sectors. This is shown when a file is, or was at one time, larger than 2TB.
  • i – Immutable: This renders the file impervious to change. The file cannot be written to, appended to, or deleted. It cannot be renamed and it cannot have a link created to it. This attribute can only be set by a superuser account.
  • I - Directory is being indexed: This attribute is not something you can set with chattr. What this indicates is that the directory is being indexed using hashed trees.
  • j – Data journaling: If the file system is mounted with the data=ordered or data=writeback options enabled then this attribute will force the data to be written to the ext3 journal before being written to the file. If the share is mounted with data=journal, than this attribute has no effect as that is the way data=journal behaves.
  • s – Secure deletion: When the file is deleted the blocks the file used are zeroed out. This attribute is not honored by ext2 and ext3 file systems.
  • S – Sychronous Updates: With synchronous update attribute set any changes to the file are immediately written to the disk. Otherwise, changes to the file are cached and then updated at a later time.
  • t- no tail merging: Tail merging, or block sub-allocation  is when a single block is used to hold the tail end of a files data. File systems are formatted into blocks for storage. The default in most Linux systems if 4KB blocks. Most files do not evenly divide into the block values resulting in the last, or tail, block of the file containing empty space. Block sub-allocation is a behavior of some file systems to aggregate the tails of multiple blocks into a single block thus freeing up the blocks those file ends would have consumed. This no tail merging attribute turns this feature off on the file.
  • T – Top of directory hierarchy: This attribute works with Orlov’s block allocator algorithm. The idea is that storing files in related directories closer together will result in faster disk access. An apt example is home directories. By grouping files with their home directories on a disk theoretically most access should be sped up. Otherwise, if directories and unrelated files are grouped together then disk access to related files will take longer. This flag will attempt to force sub-directories to be unrelated and should be spread apart.
  • u – Undeletable: This attribute sets the file to be recoverable should it be deleted, the contents of the file are actually saved. This option is not available to ext2 and ext3 file systems. This is a hold over from extfs.

A file or directories attributes can be viewed with the lsattr command. This command by itself acts like a directory listing showing all the files in the current working directory and their attributes. Specifying a single file will show only the attributes of that file. The output will look something like this:

lsattr test.txt

Produces:

————-e– test.txt

To fill in the dashes, should these values be set:

suS-iadAcj-t-e– test.txt

The lsattr command has a few flags you can pass:

  • -R – recursively list attributes of directories and their contents
  • -a – list all files in the directory including hidden files
  • -d – suppress listing contents of directories and just list directories like other files
  • -V – display version of lsattr
  • -v – display the file’s version/generation number

The version generation number is typically handled by the file system. This number is handy in networked file systems like NFS where the version number can be checked to see if a file has been changed or deleted by another user in situations where multiple users are accessing the file.

File system attributes can be manipulated by the chattr command. The command works similarly to chmod in that it requires that the change in attributes and a file or list of files be passed to it. Changing attributes can be done with:

  • + – set attribute
  • - – remove attribute
  • = – force these attributes

Thus to add the append attribute to test.txt:

chattr +a test.txt

Remember, this attribute can only be added by a superuser account like root.

To remove the append attribute:

chattr -a test.txt

Using the = to set attributes will force only those attributes to be applied. If there are already attributes that are applied to the file those will be removed if not specified by in the chattr command. The caveat here is that only those attributes that can be set with the chattr will be altered. Trying to set or remove the h attribute will fail as this can only be set by the file system not with chattr.

There are a few flags that chattr accepts. The -R flag recursively sets the attributes for all files and sub-directories below the specified or current directory.

The -V flag tells chattr to be verbose on what it is doing and will also print the program version.

The -f switch will suppress error message.

The -v switch will allow you to set the version number of a file. It requires the version number to be passed. Unless you know what you are doing, do not use this flag.

This episode of Linux in the Shell discussed extended file attributes and how to view and set them with the lsattr and chattr command.

-

bibliography:


If the video is not clear enough view it off the YouTube website and select size 2 or full screen.  Or download the video in Ogg Theora format:

Thank you very much!

Posted in Uncategorized | Leave a comment

Episode 27 – factor, primes, seq, and arithmetic

The last two episodes of Linux in the Shell discussed the use of bc as a command line scientific calculator and units for converting between different units. This episode will continue along the lines of mathematics by highlighting some other math related commands. First up is factors which, as the name implies, will output the prime factors of a given number. Prime factorization, or integer factorization, is the process of dividing a positive integer, that is greater than 1, into exact prime numbers. For instance, the following is a list of factors for positive integegers up to 20:

2 = 2
3 = 3
4 = 2 2
5 = 5
6 = 2 3
7 = 7
8 = 2 2 2
9 = 3 3
10 = 2 5
11 = 11
12 = 2 2 3
13 = 13
14 = 2 7
15 = 3 5
16 = 2 2 2 2
17 = 17
18 = 2 3 3
19 = 19
20 = 2 2 4

Basically, prime factoring involves dividing a number until all you have left is the prime factors whose product when multiplied together is the number you factored. The number 1 is not reported. The factor command, which is included in the coreutils package will factor any positive integer you provide:

factor 1382

Reports the following result:

1382: 2 691

There are only two options you can pass to factor and those are:

  • –version which will output the version of factor
  • –help which will output what factor does and these two options

The primes command takes a starting and optional stopping number and will output all the prime numbers from the start option to the stop option. If you do not provide a stop option it will continue until 4294967295 is reached or you kill the program with ctrl-c:

primes 500 1500

Primes does not have any options to pass other than the start and stop values. You must provide a start value of 0 or a positive integer. The stop value is optional but if used must be greater than the start value.

Chances are the primes command is not installed on your system by default. Primes is part of the BSD games package and should be an easy install via your package manager.
The seq command will output a complete list of floating point numbers given a starting and stopping number. You can optionally include an incremental value:

seq 5 .5 10

This example would start at the number 5 and show  numbers between 5 and 10 and increment each step by .5:

5.0
5.5
6.0
6.5
7.0
7.5
8.0
8.5
9.0
9.5
10.0

If a start number is not given seq defaults to 1. If you want to specify an increment value you must also specify a stop number. Otherwise it will treat the increment value as the stop number. If the stop number is greater than the start number the output will be null unless you provide a negative incremental value:

seq 10 1

Produces no output, but:

seq 10 -1 1

Produces a count down from 10 to 1 in 1 step decrements.

If you pass a value increments or decrements unevenly the sequence will end at the last possible number:

seq 1 .3 3

Produces the following output:

1.0
1.3
1.6
1.9
2.2
2.5
2.8

The seq command has a few options that it can take. The -s, or –separator=, will change the default separator between each number. The default is the newline (e.g.: \n):

seq -s, 1 10

Produces the output:

1,2,3,4,5,6,7,8,9,10

The -w, or –equal-width, will print all numbers with the same with padding with zeros where necessary:

seq -w 1 3 15

Produces:

01
04
07
10
13

The -f, or –format=, option allows you to specify the printf-style floating point conversion specification to display the sequence in. The accepted values are:

  • %a - hexadecimal floating point, lowercase
  • %e – scientific notation, lowercase
  • %f – decimal floating point, lowercase
  • %g – Use shortest representation: %e or %f (this is default)
  • %A - hexadecimal floating point, uppercase
  • %E – scientific notation, uppercase
  • %F – decimal floating point, uppercase
  • %G – use shortest representation: %E or %F

To change the output sequence to hexadecimal floating point:

seq -f %a 1 6

Produces:

0x8p-3
0x8p-2
0xcp-2
0x8p-1
0xap-1
0xcp-1

For more information about the output formats for seq consult the info page.

Before closing this episode there is one more BSD Games program to cover: arithmetic. The arithmetic program is a mathematics quiz that will ask you a series of math problems. To proceeded to the next problem you have to answer the current problem correctly. The game will continue indefinitely printing your score after every 20 questions. The default operation of arithmetic will use the number 0 through 10 and only addition and subtraction. There are switches to change this.

The -o switch allows you to specify the operators which can be one, more, or all of the following:

  • + – addition
  • - – subtraction
  • / – division
  • * – multiplication

Examples:

arithmetic -o +* – only addition and multiplication problems
arithmetic -o / – only division problems

The default range is 10 and this can be changed with the -r switch. Note that for addition and multiplication the range applies to the addends, terms or factors. For subtraction and division the range applies to divisor and quotient or the subtrahend and the difference.

arithmetic -r 1000

This sets the the terms range from 0 to 1000 for addition and the sutrahend and difference range from 0 to 1000 for subtraction.

Bibliography

 


If the video is not clear enough view it off the YouTube website and select size 2 or full screen.  Or download the video in Ogg Theora format:

Thank you very much!

Posted in Uncategorized | Leave a comment

Episode 26 – units

Last episode of Linux in the Shell discussed the use of the bc command to perform math on the command line. This episode continues in suit with a mathematical theme picking up from the last examples of converting between different number systems or units. While bc can help you convert between units if you know the formulas, there is another program which will do it all for you – units. Chances are units is not installed by default but a simple check in your package manager should allow you to add units to your daily tool set.

Like bc, if you execute the command units you will be placed into an interactive tool that asks you what you have:

units interactive mode

In addition to being prompted units will also show you the currency exchanges rates from date and the number of supported units, prefixes, and non-linear units. To exit units press the – keys.

On most systems units stores the unit database in /usr/share/units/definitions.units. You can verify this by executing units with the -V, or –version, switch:

verbose information

This file is plain text and can be read via a pager like less or more.

To begin converting between units simply enter the unit you have. For instance:

1 mile

You will be prompted for the unit you want:

feet

Units will then perform the conversion giving you two lines:

* 5280
/ 0.00018939394

The first line, preceeded by the “*” will give you the conversion values you requested. The second line, preceeded by the “/” indicates the inversion of the conversion. In this case, 1 mile equals 5280 feet and the inversion, 0.00018939394 miles equals one foot.

The capacity of units is very large and you can easily determine what conversions are applicable to a given unit by entering “?” for “you want:”. This will generate a list of applicable units you can convert to.

The functionality of units is quite simple but the implementation can be tricky. For instance, converting between fahrenheit and celsius is not as straight forward as meters or feet:

you have: tempfahrenheit(32)
you want: tempcelsius

The value returned is 0 degress celsius. You could also use:

you have: 0 degC
you want: degF

But the value returned is 0 degree fahrenheit. How is this so?

you have: tempfarenheit(212)
you want: tempcelsius

This returns a value of 100 temp but:

you have: 212 degF
you want: degC

Returns a values of 117.77778. That does not seem right. The expectation that presenting the units in these ways indicates they mean the same thing. That tempC and degC are going to return the value of the temperature conferted from fahrenheit. While the first unit conversion using tempcelsius does this, degC on the other hand is showing the temperature change in celsius given the temperature change in fahrenheit. That is, a 212 degree fahrenheit change in temperature is equivalent to 117.77778 degree celsius change in temperature. Understanding this is something you have to pick up reading the man page. The units definition page will not easily convey this information but it does provide a wealth of knowlege on the units and how they were derived.

Similar to temperature conversions, converting between megabytes, gigabytes, and kilobytes is not as evident as you would expect. While most versions of units have tabbed completion, if you attempt to complete meg you will not megabyte but just meg. This is equivalent to megabits and will not give you the value expected if you attempt to convert 15gb into kilobytes. Instead you must use the proper binary prefixes:

  • KiB = kilobyte
  • MiB = megabyte
  • GiB = gigabyte
  • TiB = terabyte
  • PiB = Petabyte
  • EiBxxxxx = Exabyte

To convert 15 gb into kilobytes:

you have: 15 GiB
you want: KiB

Produces the value: 15728640

Again, if you get stumped in what you are looking for consult the definitions.units file. A quick search will probably turn up the answer.

Units has a few command line switches:

-q, –quiet, and –silent suppress the prompting of the interactive tool. You will not see the “you have:” and “you want:” prompts.

The –compact supress the printing of the “*” and “/” but still show both values.

-l, –one-line, will only show the first line of output and surpress the inverse of the conversion.

The preceeding three switches are combined in the -t, –terse, switch which is equivalent to –quiet, –compact, and –one-line. That is, no prompts and only the conversion line is shown.

You can execute the units command to perform conversions without having to go into interactive mode by providing the request on the command line like so:

units “15 feet” inches

Produces the output:

Currency exchange rates from 2012-10-24
* 180
/ 0.0055555556

You can use the -t switch to just get the conversion you are looking for:

180

The -v, or –verbose, switch will provide a bit more output than normal:

units -tv “15 feet” inches

Produces the output:

Currency echange rates from 2012-10-24
15 feet = 180 inches

This entry just touched the surface of the units command. Units is an incredibly powerful tool at your disposal. To learn more about the command consult the man and info pages and consider giving the definitions list a perusal.

  •  man units
  • info units

 

If the video is not clear enough view it off the YouTube website and select size 2 or full screen.  Or download the video in Ogg Theora format:

Thank you very much!

Posted in Uncategorized | Leave a comment

Episode 025 – bc

Math from the Linux command line is one of those tasks that is not as straight forward as you may think. There are many tools that will allow you to perform mathematical functions accessible to you, but to perform simple arithmetic is not as simple as just entering some equation. You can use the echo command to perform basic mathematical problems but it does not allow for decimals making division in particular problematic. But if you want to simply add, subtract, multiply, or divide some values it is a handy resource:

echo $((512 * 1024))

That should return the value

524288

But,

echo $((4/3))

Returns the value of:

1

When it should be closer to:

1.333333

If you are going to use echo to perform simple arithmetic you need to follow the syntax:

echo $(( number symbol number))

Similarly the expr command can be used for simple arithmetic:

expr 4 + 7

Returns the result

11

But again, expr will not work well with decimals:

expr 4 \/ 3

Returns:

1

Note in this example the “/” had to be escaped.

To perform more complex mathematical operations from the command line use the bc tool. GNU bc is “an arbitrary precision calculator language” that will accept input from files, redirection or will run it’s own interactive session. Issuing bc by itself puts you into interactive mode:

screen shot 1

interactive bc session

interactive bc session

This is the same as executing bc with the -i, or –interactive, switch. At this point you can enter the statement:

4/3

Which will return

1

We seem to be in the same predicament as before, no decimals. This is because by default bc is operating with a scale of 0. Scale is a special variable in bc that determines the total number of decimal digits after the decimal point. To see the current value of scale while in bc execute:

scale

The default return should be

0

You can alter scale like so:

scale=3

Which sets scale to 3 decimal digits after the decimal point. So now:

4/3

Returns

1.333

To exit bc simply type quit.

There are a few other switches to the bc tool. The -q, –quiet, switch does not print the welcome message when entering into interactive mode. The -v, or –version, switch will show the version and copyright information for bc and then quit.

The -l, –mathlib, will load the standard math library when bc is executed. This sets the default scale to 20 and allows for additional math functions:

  • sine – s(x), x in radians
  • cosine – c(x), x in radians
  • arctangent – a(x), arctangent returns in radians
  • logarithm – l(x)
  • exponential function – e(x)
  • bessel – j(n,x)

Aside from evoking bc interactively, bc can operate on redirection:

echo 4/3 | bc

Returns

1

Again, we are missing the decimal point. There are to ways to resolve this:

echo 4/3 | bc -l
echo “scale=3; 4/3″ | bc

Which return respecitvely

1.33333333333333333333
1.333

Recall that the -l switch loads the math library in and sets scale=20 by default, so there are 20 decimal digits after the decimal point. The second example we set the scale variable to 3 and thus returns 3 decimal digits after the decimal point.

There are three other special variables that alter the behaviour of bc:

  • ibase = Set the base value of the input numbers, default 10 (decimal)
  • obase = Set the base value of the output numbers, default 10 (decimal)
  • last = Is set to the value of the last number printed

Both obase and ibase are set to decimal value by default. To switch to output in binary:

obase=2

To switch to input in Octal:

ibase=8

Note that changing this values will set the ibase and obase for all subsequent statements. To restore back to defaults either set each back to decimal:

obase=10
ibase=10

Or quit and restart bc. Issuing these as a redirection to bc will only set those values for that session:

echo “obase=2; 5*5; obase=10; last” | bc

Will return:

11001
25

The standard order of operations applies to the bc tool and can be altered with the use of parenthesis:

echo “5+2/2″ |bc -l
echo “(5+2)/2″ |bc -l

Return

6
3.50000000000000000000

The bc tool has it’s own language called bc. You can do far more than just calculator functions. You can define your own functions with the define command. For example:

define litsconverter (a) {
obase=2;
print “binary value is “;a*1; print “\n”;
obase=8;
print “octal value is “; a*1; print “\n”;
obase=16;
print “hex value is “; a*1; print “\n”;
obase=10;
print “all finished (p.s.: Ignore the 0 it returns at the end of every function if a return is not used”;
}

Now running:

litsconverter(15)

Will convert 15 (or whatever number you put in there) to binary, octal, and hex values the returns obase to decimal. The litsconverter function will remain available until you quit bc. Also note, that if we did not set obase back to decimal it would have stayed at hexidecimal until changed later on or you exited bc.

There is a lot more functionality to bc than just listed here. Consult the man page to get started on how to create your own functions to unlock the power of bc.

 


If the video is not clear enough view it off the YouTube website and select size 2 or full screen.  Or download the video in Ogg Theora format:

Thank you very much!

Posted in Uncategorized | 1 Comment

Episode 024 – time and /usr/bin/time

The time program is a handy tool to not only guage how much time in seconds it takes a program to run, but will also display how much user CPU time and system CPU time was used to execute the process. To understand these values you must grasp how the kernel handles the time reporting for the process. For example, the output of:

time ls

is

real 0m0.007s
user 0m0.000s
sys 0m0.003s

The first line, real, is how much real time or “wall-clock” time the process took to execute. In this case, 0.007 seconds or micro-seconds. The second line, user, is how much user time the CPU used. User time is amount of time the CPU spends performing an action for a program that is not a system call. The third value, “sys” referred to as system CPU time, is the amount of time the CPU spends peforming system calls on behalf of the program. A system call is a request made to the Kernel by the program. Therefore, CPU time is the total of user time plus system time. CPU time will not necessarily equal real (wall-clock) time, and if it does not total CPU time would be less than wall-clock time. There can be many reasons for the discrepancy between wall-clock and CPU time including the process waiting for another process to complete. In the example above total CPU time is 0.003 seconds where as wall-clock time is 0.007s.

Most Linux distributions default to using the Bash shell. Bash has a built in time function that operates as listed above and is quite different than the actual time command. The functionality of the bash time command is very limited to the actual time command. The remainder of this article will focus on the actual time command as opposed to the built in bash time command.

Most distributions have the time command installed under /usr/bin. Executing:

which time

Should output the path to the time command:

/usr/bin/time

If no value is returned chances are you do not have the time command installed (I had this issue on Arch and had to install the time command by sudo pacman -S time, on Slackware it was already installed).

To use the time command as opposed to the bash built in command you must provide the full path to time:

/usr/bin/time ls

Note that the output is quite different:

0.00user 0.00system 0:00.00elapsed 50%CPU \
(0avgtext+0avgdata 784maxresident)k
0inputs+0outputs (0major+250minor)pagefaults 0swaps

There is a lot more information provided compared to the builtin Bash time command.

The first three items: 0.00user 0.00system 0:00.00elapsed should look familiar. These values are user CPU time, system CPU time, and wall-clock time respectively. To understand the remaining values better focus will be directed on how to actually format the output of the time command. This is done with the -f, or –format=, switch followed by the formatting strings. Similar to the date command, the formatting strings are in the form of “%” + variable:

  • %E = Elapsed wall-clock time in [hours:]minutes:seconds – note hours will not show unless the process does take over an hour
  • %e = Elapsed wall-clock time in seconds only (this option is not available in tcsh).
  • %S = system CPU time time in seconds
  • %U = user CPU time in seconds
  • %P = Percentage of CPU that was given to the process – equation is (%U + %S) / %E
  • %M = Maximum resident set size of the process during its lifetime in KB. This value is the maximum amount of physical memory (RAM) this process utilized during the life of the process.
  • %t = Average resident size of the process in Kilo Bytes. This is the average amount of physical RAM the process utilized over its lifetime (not available in tcsh).
  • %K = Average total (data + stack + text) memory used by the process in Kilo Bytes. In a nut shell
    • data = Global data, variables, etc.
    • stack = Where variables are declared and initialized
    • text = The actual programs
  • %D = Average size of the processes unshared data area in Kilo Bytes – This is where data for the process is stored.
  • %p = Average size of the processes unshared stack space in Kilo Bytes – This is where variables are declared and initialized (not in tcsh)
  • %X = Average size of the processes unsahred text space in Kilo Bytes – This is where the actual program resides
  • %Z = System’s page size – This is the size of a single block of contiguous virtual memory (likely to be 4096 bytes) (not in tcsh)
  • %F = Number of major page faults that occurred while the process was running. A major page fault is when the process attempts to access a page that is mapped in the virtual address space but is not available in physical memory, it needs to find a space in physical memory to map the page.
  • %R = Number of minor page faults that occurred while the process was running – Similar to a major fault, a minor page fault is when the page is actually loaded in memory at the time but is not marked in the memory management unit as being loaded in memory. The operating system needs to mark the page as loaded in the memory management unit before it can be loaded.
  • %W = The number of times the process was swapped out of main memory
  • %c = The number of times the process was context-switched involuntarily. In a multi-processing environment a process may need to be switched out for another process that needs the CPU. The CPU state is saved so the process can be resumed. Involuntary swapping can be caused by the application being forced to swap out because it’s time slice (a lotted time to execute) was exceeded.
  • %w = The number of waits that the program was context-switched voluntariliy. A volunary swap may be because the process was waiting for another process or I/O action to complete before it could continue.
  • %I = Number of file system inputs by the process
  • %O = Number of file system outputs by the process
  • %r = Number of socket messages received by the process
  • %s = Number of socket messages sent by the process
  • %k = Number of signals delivered to the process
  • %C = Name and command-line arguements of the command being timed (not in tcsh).
  • %x = Exit status of the command (not in tcsh).

Now that these values have been defined the output of:

/usr/bin/time ls

From above can be further explained:

0.00user 0.00system 0:00.00elapsed 50%CPU (0avgtext+0avgdata 784maxresident)k 0inputs+0outputs (0major+250minor)pagefaults 0swaps

In a nutshell:

%U %S %E %P (%X + %D %M)k
%I + %O (%F+$R)pagefaults %W

(User CPU Time) (System CPU Time) (Wall-Clock Time) ( (Average size of shared text space) + (Average size of unsahred data are) (Maximum Resident Set Size of Process) )k
(Number of File System Inputs) + (Number of File System Outputs) ( (Number of Major Page Faults) + (Number of Minor Page Faults) pagefaults (Number of times process was swapped out of memeory)

As stated, formatting of the /usr/bin/time command can be done with -f, or –format=, switch followed by a string including the above variables:

/usr/bin/time -f “%e may not equal %S + %U” ls

The output of this may look something like:

0.04 may not equal 0.00 + 0.00

To emulate something like the output of the Bash built in time command you could execute:

/usr/bin/time -f “real\t%e \nuser\t%U \nsys\t%S” ls

The output may look like:

real 0.04
user 0.00
sys 0.00

Or you could pass the -p, –portability, switch which provides the same formatting:

real %e
user %U
sys %S

The output of time can be directed to a file with the -o, or –output=, switch and supplying the file name:

/usr/bin/time -o times.txt ls

This would redirect the output of the /usr/bin/command to the file times.txt. If you executed the same command again, it would overwrite the original times.txt file. If you want to preserve the original contents of the times.txt file use the -a, or –append, switch with the -o, –output=, switch:

/usr/bin/time -o times.txt -a ls

The final switch to discuss is the -v, or –verbose, switch which will output all the values possible that /usr/bin/time can report regarding a process.

/usr/bin/time -v tar cvf test.tar.gz lib

Produces the following output:

output of /usr/bin/time -v

output of /usr/bin/tiime -v

That is the time and /usr/bin/time commands in a nutshell. Remember, if you want the full feature set of the time command you must provide the full path to the time command. Otherwise you will more than likely use the built in time command of the shell you are running.

Video failed youtube processing, will be up shortly, use archive.org link below.

If the video is not clear enough view it off the YouTube website and select size 2 or full screen.  Or download the video in Ogg Theora format:

Thank you very much!

Posted in Uncategorized | Leave a comment

Episode 023 – date

The date command will not only display or let you change the current date and time but is the go to utility for getting date and time information into scripts. Evoked by itself the date command will output the current system date based upon the rules of the LC_TIME format. The LC_TIME format defines the rules for formatting dates and times. LC_TIME is a subset of locale which defines the overall environment based upon the chosen language and cultural conventions. You can see the current LC value by issuing the locale command. You can see time specific information for your system by issueing:

locale -m LC_TIME

This will list the available locales configured for your system. Configuring of the locale values is dependent upon the distribution of Linux you are running. For instance, in Arch Linux you can set the locale by editing the /etc/locale.conf file. On a RedHat based system this value is store in /etc/sysconfig/i18n. On a Debian based system the default locale is stored in /etc/default/locale. Managing locale information is beyond the scope of this article so consult your distrobution’s documentation should you need to change your locale values or set a new default.

For the purpose of this article examples will be shown from the LC_TIME=en_US.UTF-8 value.

The date command issued by itself reports the date in the following format (Abbr stands for abbreviated):

[Abbr Day of Week] [Abr Month] [Month Day]  /
[Hour:Minute:Second] [Timezone] [Year]
Sat Feb 9 11:48:51 EST 2013
%a %b %d %H:%M:%S %Z %Y

The last row is the relates the format controls to date corresponding to the equivalent values displayed on the second line. These will be covered shortly.

The -d, or –date=, parameter takes a string format for a date and returns a value as opposed to the current date. There are many different possible strings to pass for instance:

date -d “2 days”

will return the date string like above but 2 days from the current time:

Mon Feb 11 11:48:51 EST 2013

Issuing:

date -d “2 days ago”

Will return the date string above but from 2 days ago:

Thu Feb 7 11:48:51 EST 2013

Now realize that the actual time would be the time from 2 days prior or 2 days hence from when the date command was run and would not output the exact time string as the first entry. The example was given just an example based off the first date above.

Some of the values you can pass with the -d command are:

  • days
  • months
  • years
  • hours
  • minutes
  • seconds
  • ago

The -f, or –file=, switch operates the same way as the -d but will read a file where each line is a date value in the format of -d. For instance, if there was a file called datefile.txt with the following entries:

3 days ago
1 month
2 hours 9 minutes
Sun

Then:

date -f datetime.txt

Would read each line of the file and output the corresponding values:

Wed Feb 6 12:03:49 EST 2013
Sat Mar 9 12:03:49 EST 2013
Sat Feb 9 14:12:49 EST 2013
Sun Feb 10 00:00:00 EST 2013

The date command will allow you to view the last modification date of a file with the -r, or –refrence=, switch:

date -r datetime.txt

Returns the time the file was last modified:

Sat Feb 9 12:03:45 EST 2013

Before we look at the remaining swtiches to date, let us turn to how the output of date can be formatted differently. The syntax for fomatting the date ouput is as follows:

+%?%?%?

Where ? is a placeholder for a variable. Some of the more common variables are:

  • %a = locale’s abbreviated week day – Mon, Tue, Sun
  • %A = locale’s full weekday name – Monday, Tuesday, Sunday
  • %b = locale’s abbreviated month name – Jan, Feb, Aug, Dec
  • %B = locale’s full month name – January, February, August, December
  • %c = locale’s date and time – Sat Feb 9 12:12:45 2013
  • %C = Century – 21
  • %d = day of the month with leading 0
  • %D = date in the locale format – 02/09/13
  • %e = day of the month with space padding instead of leading 0
  • %F = full date which – 2013-02-09 – equivalent to %Y-%m-%d
  • %H = 24 hour with leading 0 – 00, 08, 12, 18
  • %I = 12 hour with leading 0 – 01, 08, 10, 12
  • %j = day of the year with leading zero’s – 001, 015, 115, 350
  • %m = month with leading 0 – 01, 05, 12
  • %M = minute with leading 0 – 01, 08, 15
  • %p = locale’s equivalent of AM or PM
  • %P = locale’s equivalent of am or pm (note same as %p but lower case)
  • %s = seconds since 1970-01-01 00:00:00 UTC
  • %S = seconds with leading 0
  • %u = day number of day of the week – 1 = Monday, 7 = Sunday
  • %U = week number of the year with Sunday as the first day of the week – 00-53
  • %V = week number of the year with Monday as the first day of the week (ISO week number) – 01 – 53
  • %w = day number of day of the week with Sunday as the first day and numbering from 0 to 6 – 0 = Sunday
  • %W = week number of the year with Monday as teh first day of the week – 00-53
  • %y = last two digits of the year
  • %Y = year as four digits
  • %z = numerical timezone – -0500
  • %Z = alphabetic time zone appbreviation – EST, EDT, PST

The output of most formats is to pad numeric fields with zeros. This can be altered with the following flags:

  • - (hyphen) – do not pad the field – 09 becomes 9
  • _ (underscore) – pad with spaces – 09 becomes ” 9″ (double quotes added to identify space)
  • 0 – pads with zeros – 8 becomes 08
  • ^ – use upper case if possbile – +%^P (pm) becomes +%p (PM)
  • # – use lower case if possible – +%#p (PM) becomes +%P (pm)

Appending a number between the % and variable will specify the width of the field:

date +%Y
date +%2Y
date +%8Y

The following values are returned:

2013
2013
00002013

Note that the second entry date +%2Y did not truncate the date to two characters. Field width will not limit the output but will pad the output.

If your locale supports it there are two more flags that can be applied to some of the format options:

  • E – applied to %c, %C, %x, %X, %y, and %Y – %EY – this will use the locales alternate representation of the value. 
  • 0 – can only be applied to numeric conversions and will use the locale’s alernate numeric system.

To put some of this in perspecitve given our original date output: Sat Feb 9 11:48:51 EST 2013, the following lists the output returned by the specific formatting:

date +%Y = 2013
date +%Y%m%d = 20130209
date +%Y-%m-%d = 2013-02-09
date +%P = pm
date +%w = 6
date +%W = 05
date “+Today is %A have a good day” = Today is Saturday have a good day

Note that the third and last examples add extra content into the format. Any non-variable will be echoed back out in the string. If you want to include a “%” then issue like so: %%:

date “+%A is the 100%% best day”

There are other format tags for date:

  • -R, –rfc-2822 – Sat, 09 eb 2013 14:20:43 -0500
  • -I, –iso-8601= – 2013-02-09
  • –rfc-3339=seconds – 2013-02-09 14:20:43-0500

These three values format the date and time according to their respective specifications. The first two, -R and -I show their defaults and require no further specification. –rfc-3339 requires that you specify a TIMESPEC. You can also specify TIMESPEC for –I using –iso-8601=TIMESPEC. TIMESPEC values between the –iso-8601 and –rfc-3339 differ:

  • date, –iso-8601 and –rfc-3339, just prints date – 2013-02-09
  • hours, –iso-8601 only, prints date and hours – 2013-02-09T14-0500
  • minutes, –iso-8601 only, prints date, hours and minutes – 2013-02-09T14:40-0500
  • seconds, –iso-8601 and –rfc-3339, prints date, hours, minutes and seconds – 2013-02-09T14:40:05-0500 or 2013-02-09T14:40:05-05:00
  • ns (nano-seconds), –iso-8601 and –rfc-3339, prints date, hours, minutes and nano-seconds – 2013-02-09T14:40:05,454325891-0500 or 2013-02-09T14:40:05,454325891-05:00

The -u, or –utc or –universal, flag will output the date in Coordinated Universal Time

date
date -u

Shows the comparison:

Sat Feb 9 14:46:17 EST 2013
Sat Feb 9 19:46:19 UTC 2013

The date command can not only display the date or time but can also set the date and time with the -s, or –set, flag. This of course requires elevated priveleges:

date -s 2013-01-15

This would set the system date and time to:

2013-01-15 00:00:00 EST 2013

But:

date -s “2013-01-15 08:15:44″

Would set the date to:

2013-01-15 08:15:44 EST 2013″

The date command is a fantastic tool available for providing date stamps in your scripts. For example:

#!/bin/bash
if [ -n "$1" ] && [ -e "$1" ]
then
cp $1 $1.`date +%Y%m%d-%H%M%S`
fi

If you save this script as makebkup, make it executable, when called this script will take a file as input using the cp command make a backup using date to create a file name with a time stamp:

makebkup mypoem.txt

This would effectively run:

cp mypoem.txt mypoem.txt.20130209-150823

The date command is a powerful tool to have at your disposal. Check out the man or info pages for the full list of formatting variables.

 


If the video is not clear enough view it off the YouTube website and select size 2 or full screen.  Or download the video in Ogg Theora format:

Thank you very much!

Posted in Uncategorized | Leave a comment