Posted by & filed under Linux.

We all know the humble ping command. Check out its nice, human-readable output:

$ ping -c 4 google.com
PING google.com (74.125.225.131) 56(84) bytes of data.
64 bytes from ord08s09-in-f3.1e100.net (74.125.225.131): icmp_req=1 ttl=63 time=26.6 ms
64 bytes from ord08s09-in-f3.1e100.net (74.125.225.131): icmp_req=2 ttl=63 time=27.4 ms
64 bytes from ord08s09-in-f3.1e100.net (74.125.225.131): icmp_req=3 ttl=63 time=25.7 ms
64 bytes from ord08s09-in-f3.1e100.net (74.125.225.131): icmp_req=4 ttl=63 time=26.8 ms

--- google.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3006ms
rtt min/avg/max/mdev = 25.723/26.686/27.481/0.641 ms

Now, imagine you want to parse this data so that your application can read it. Here’s a regular expression I nabbed from a Stack Overflow answer which grabs information about the individual ping responses:

^(?<Size>\d+) bytes from (?<DestinationHost>[^\s]+)
 \((?<DestinationIP>.+?)\): icmp_seq=(?<ICMPSequence>\d+)
 ttl=(?<TTL>\d+) time=(?<Time>.+)$

On the surface it seems like a perfectly acceptable solution. Until, that is, the maintainer of the app changes the formatting/wording of the output. Not to mention the slight overhead of running a regex, or even the effort it takes for the app to take this output, make it pretty for a human, just to have us convert it back into a format for machine consumption. But really, the biggest overhead IMO is the time it takes to build a script, regex, or grep command to parse such output.

JSON Output

Now, imagine a world where a large subset of Linux command line utilities have a magical –json (or –xml, –yaml) option, which takes that normally human-readable output displayed via stdout and instead renders it as the specified data transmission language. Here’s an example of our previous ping command converted to JSON:

$ ping -c 4 google.com --json
{
    "target": "google.com",
    "target_ip": "74.125.225.131",
    "bytes_of_data": 56,
    "statistics": {
        "packets": 4,
        "received": 4,
        "packet_loss": 0.0,
        "time": 3006,
        "min": 25.723,
        "avg": 26.686,
        "max": 27.481,
        "mdev": 0.641
    },
    "transmissions": [
        {
            "size": 64,
            "hostname": "ord08s09-in-f3.1e100.net",
            "address": "74.125.225.131",
            "icmp_req": 1,
            "ttl": 63,
            "time": 26.2
        },
        {
            "size": 64,
            "hostname": "ord08s09-in-f3.1e100.net",
            "address": "74.125.225.131",
            "icmp_req": 2,
            "ttl": 63,
            "time": 27.4
        },
        {
            "size": 64,
            "hostname": "ord08s09-in-f3.1e100.net",
            "address": "74.125.225.131",
            "icmp_req": 3,
            "ttl": 63,
            "time": 25.7
        },
        {
            "size": 64,
            "hostname": "ord08s09-in-f3.1e100.net",
            "address": "74.125.225.131",
            "icmp_req": 4,
            "ttl": 63,
            "time": 26.8
        }
    ]
}

Suddenly, we have an amazingly simple output format for parsing! Simply fire up your language of choice’s JSON library and parse that output. We could even possibly take the contents of stderr and display it in the same output, but in an error node.

Pitfalls

I’m not sure of a good solution for handling ‘streaming’ output, e.g. a ping command without the -c option. Perhaps we could actually display several complete JSON documents, maybe requiring a delimiter of some sort , but at this point we are no longer dealing with pure clean JSON and are instead starting to develop a JSON superset.

This also wouldn’t make a lot of sense to be used with interactive commands, or really anything that needs a Ctrl+C when complete. So, for this reason, it would probably mean that –json is limited to being used in conjunction with other arguments, and that it cannot be used with all CLI apps.

Arguments Against

What if the author of the app changes the structure of the JSON document? The applications digesting the output of the app would still need to be re-written, but grabbing a different data node is usually easier than getting familiar with the new output and writing a different parser.

Why use JSON? Well, JSON is a pretty commonly used data format (at least on the web, it isn’t too popular among Linux utilities). Importantly, JSON can be easily converted into different formats such as XML or YAML. One could write a utility and pipe the output through it, such as “ping -c 4 google.com –json | json2xml”. Or, even better, the author of the app could provide switches for the different types of data.

What about the Linux philosophy of having simple, human readable output? We would still keep the default, human readable output. The –json flag would simply be an option that the maintainer of the app would add as an enhancement.

My app already has a –json switch for doing something else. The –json flag is just an example, really. There doesn’t need to be a universal switch for enabling this output (although it would be nice). The overall goal of these switches would be to make parsing of the output of the command built into the actual app instead of being external.

Inspiration

Of course, I wasn’t inspired by the ping command to write this (with its simple output), but instead the output of the command iwlist wlan0 scan. I’ve been writing a parser using node.js, and the code is a bit scary looking (as I feel most parsers must be). Here’s an example of the output being parsed (the Cell’s repeat once for each network):

wlan0     Scan completed :
          Cell 01 - Address: 40:F4:EC:7F:A1:52
                    Channel:11
                    Frequency:2.462 GHz (Channel 11)
                    Quality=70/70  Signal level=-31 dBm
                    Encryption key:on
                    ESSID:"CTL_Wireless"
                    Bit Rates:6 Mb/s; 9 Mb/s; 12 Mb/s; 18 Mb/s; 24 Mb/s
                              36 Mb/s; 48 Mb/s; 54 Mb/s
                    Mode:Master
                    Extra:tsf=00000203d78a11f8
                    Extra: Last beacon: 13986ms ago
                    IE: Unknown: 000C43544C5F576972656C657373
...truncated...
                    IE: Unknown: 2D1A2C181BFFFF000000000000000000000000000000000000000000
                    IE: IEEE 802.11i/WPA2 Version 1
                        Group Cipher : TKIP
                        Pairwise Ciphers (1) : CCMP
                        Authentication Suites (1) : PSK
                    IE: Unknown: 3D160B000700000000000000000000000000000000000000
                    IE: Unknown: 9606004096001400
                    IE: WPA Version 1
                        Group Cipher : TKIP
                        Pairwise Ciphers (2) : TKIP CCMP
                        Authentication Suites (1) : PSK
                    IE: Unknown: DD180050F2020101800003A4000027A4000042435E0062322F00
...truncated...
                    IE: Unknown: DD1D0040960C032A6F6EFA818036010000130604000000FDB8F23A2A1

Thanks

Thanks for giving this a read! If you know of any possible solutions to the ‘streaming’ output, if you can think of a better output format, if you know of a similar initiative, or even if you think I’m an idiot, please let me know in the comments below.

Update

Thanks for all of the great feedback on Reddit (and even a little bit on Hacker News). It is interesting to see how much of a controversial subject this is. One one side, there are the *nix purists who have been writing programs for years, and have always understood the plain-text Unix philosophy. On the other hand, there are the (probably younger) half who love the thought of being able to easily read the output of commands without the need for complex parsing instructions.

I think the biggest issue is the purists hating the thought of JSON; the new-comer / web-centric data format I mentioned in the title and ping output example. Again, that was just an example language. Most of us just want a standardized method for parsing command output, and surprisingly, we really don’t care what that format is, as long as it is easy to work with.

The best example someone had of this type of thing already being in the wild is the emacs formatted output for the ls command, as pointed out by ISV_Damocles on Reddit. If there is an output format for the most commonly used Unix command just to be consumed by one program, why not have a standardized output option easily digested by any program?

Thomas Hunter II

Support Thomas on Patreon. Thomas is the author of Advanced Microservices and a prolific public speaker with a passion for reducing complex problems into simple language and diagrams. His career includes working at Fortune 50's in the Midwest, co-founding a successful startup, and everything in between.

Tags: