Plotting with ASCII Timestamps

I use EZL more than anyone I know. As an engineer I have more than my fair share of data to view. Generally my data are in nicely formatted columns with plottable, numerical, horizontal coordinate (abscissa) values. This is true because I am usually the one who created the data in the first place.

On more rare occasions though, I will have to look at data where the abscissa values are in a dreaded ASCII timestamp representation, such as “1:12:20 AM, 1/4/02” or even “Jan 03, 2014 11:45:23 PM”. Such was the case earlier today. My first attempt to plot the data failed, however, and a small EZL command error on my part had me scratching my head for a few moments. So I thought “If I can make this mistake, I bet it’s a common source of frustration for others as well” (to paraphrase myself). And so, a blog was born.

The input data came from a little battery operated temperature logger. Often the first thing I do before plotting a file with complex plotting parameters is to view the file in a file editor. So, I right-clicked the file, then selected View. As shown below, I have my default file editor set to Notepad++ (note, you can change your default editor by clicking Default Settings within the System toolbox. Then enter the editor path in the “Data file editor” box. I have mine set to “C:\Program Files (x86)\Notepad++\notepad++.exe”. This ONLY changes your EZL default editor – it does not change your Windows default editor, so feel free to play with these settings.)

Looking at the file, I know I want to plot columns 2, 3, and 4 (temperature, humidity, and dew point) against the timestamp, so my EZL command should look something like this:

ezl –f temperature_logger_0510929005.txt –y 2 3 4 –date 7 –time 5 –header 3

The –f specifies the input file and the –y specifies which columns to plot. We’ve seen these many times. What’s immediately different here, however, is that I have no –x to specify the x-axis column. Instead I have replaced –x with a pair of –date and –time columns. So far, so good. But how do I specify the format for the time and date data? How does EZL know whether to expect “Jul 16, 2014” or “7/16/2014”? Actually, it doesn’t. If we don’t tell EZL how the timestamp is formatted it will assume these formats: “2014-07-16” and “13:44:10” (or, “%Y-%m-%d” and “%H:%M:%S”) for the date and time, respectively, which is clearly wrong in both cases for our particular input file.

Because I don’t plot ASCII time-stamped data often, I decided to use EZL’s File Setup interface. So, I dragged the file onto the empty plotter and began to setup with the file:

Formatting the Time

In the image above, I have highlighted the values which I modified from their default states. First I set the third column to “y2” from the column’s dropdown box, and set the fourth, fifth, and seventh columns to “y3”, “time”, and “date”, respectively. I set the Header row to “Row 3” from the Other Options section, so that EZL will read in the labels from row 3 to use for the legend. Then, I selected the “time” tab (which appears only after a time column has been designated) from the Field Definition section, and changed the default text from “%H:%M:%S%F” to read “%H:%M:%S AM”. There are a few noteworthy points to make here:

First, notice* that the Field Definition section always shows an example result of the selected (or manually entered) format specifiers. The example here shows that our “%H:%M:%S AM” specifiers would result in a time resembling “08:30:00 AM”.
*And just in case you didn’t notice – I pointed it out in the image with a “Don’t forget to use this!” callout.

Second, recall that the default was “%H:%M:%S%F”. The %H, %M, and %S specifiers are obvious enough – hours, minutes, and seconds… but what’s with the %F? The %F specifiers allows for fractions of a second. For instance, the example display for the default specifiers would show “08:30:00.000”, with the .000 resulting from the %F specifier. It turns out, that there is no harm in leaving the %F specifier in place, even when the input timestamps do not include fractional seconds.

Finally, notice that my custom format, namely “%H:%M:%S AM”, has this strange “AM” portion. Clearly it’s an attempt to instruct EZL that we’re expecting 12-hour time with AM/PM indicators, but it’s not quite a specifier, and it’s not quite a fixed value – so what is it? In fact, it’s just a placeholder and any two characters would suffice, so long as they are non-specifier and non-delimiter characters. When parsing the time fields, EZL automatically searches for AM/PM indicators and, if found, assumes the timestamp is in 12-hour time and adjusts it to 24-hour time accordingly. The placeholders therefore, simply extend the time fields to include the additional characters. For instance, with a time specifier of “%H:%M:%S” the first time value (from row 4 of the image) would pass only “1:44:10” as the first time field, whereas “%H:%M:%S AM” (or, equivalently “%H:%M:%S xx” ) would pass “1:44:10” followed by a space, followed by two additional characters, resulting in the complete desired field “1:44:10 PM” for the data in row 4. Notice that our particular data have spaces between the times and the AM/PM indicators which push the AM/PM indicators to a new column (column 6) as shown in the image. To accommodate for this, our specifier must also include a space (in this case between the %S and the AM/PM placeholder). The space in the specifier therefore tells EZL that the time fields begin at column 5 and extend to column 6.

Formatting the Date

Satisfied that the time column was formatted properly, I then switched to the “date” tab and selected the specifier format shown below:

When I clicked “Ok” to plot the results, I was confronted with an empty plot and EZL provided this warning statement:

> Warning: Date/time parser unable to convert value.
           Format specifier : %H:%M:%S AM %m/%d/%y
           Input value      : 05:00:45 PM 07/22/2014
> Total date/time parser warnings: 9036

Can you see where I went wrong? Admittedly, it took me a few moments… I had accidentally chosen the wrong specifier for the year field. I chose “%y” (lowercase y) which designates a two-digit year such as “14” instead of “%Y” (uppercase y) which designates a 4-digit year such as “2014”. Had I been more careful and checked my format against the example results displayed in the Field Definition section, I could have saved myself some confusion.

To fix the problem, I simply pressed the Up-Arrow on the keyboard to recall the command which was issued by the File Setup tool, and manually edited the command, changing the %y to %Y. Pressing Enter then produced the desired results:

And we have our plot!

Notice that even though the timestamps were read in as ASCII formatted strings,  EZL automatically converts these to Modified Julian Date (MJD) representation. If you wish to have the plot’s x-axis tick labels displayed as Date/Time* instead of MJD, simply expand the Grid Adjustments toolbox and click “MJD -> Date/Time”.
*Note – when displaying Date/Time tick labels, the time portion will be masked out if all grid lines happen to fall on day boundaries (i.e. 00:00:00). As you zoom in, the times will reveal themselves as necessary.

Try It Yourself!

I have posted the file discussed in this blog here: Temperature data logger
Feel free to download it and try this out for yourself.

Alternatively, you can use EZL’s Web Interface tool and plot it directly from the web! Try this (you must have version 1.1.1 or later. Please Download an updated copy, if you have 1.1.0 or earlier.):

1. Go to Tools -> Web Interface
2. In the Address box, type (or copy/paste):  www.ezlsoftware.com/tutorial/temperature_logger_0510929005.txt
3. Change the Y-Axis input box from 1 to: 2  3  4
4. Click the Advanced Setup button
5. Uncheck Append data mode
6. Check “Use date column” and enter
   • Column: 7
   • Format: %m/%d/%Y
7. Check “Use time column” and enter
   • Column: 5
   • Format: %H:%M:%S AM
8. Check “Use header row” and enter
   • Row: 3
9. Click Ok to close the Advanced Setup window
10. Back in the Web Interface window, check the Link to Plotter box
11. Click Connect

Assuming EZL has access through your firewall, you should now see the contents of the data file displayed in your Web Interface, and three curves shown in the plotter!

Blair Fonville is the Co-Founder of EZL Software, LLC., a developer of scientific plotting software and data analysis tools for engineers, scientists, students, and time lab professionals.