tracert2csv – Format tracert output into comma-delimited CSVs

Part of the communications architecture for this research project I’m working on depends on TTL (time-to-live) of data as it travels through various networks.

tracert is the perfect tool for such an application; it’s just a shame that there’s no friendly way to generate and output analytics-ready data.

tracert2csv takes tracert output and processes it into a comma-delimited CSV document for easy import into spreadsheet software such as Excel, databases like MSSQL, or analytics tools.

Check out the project on Github; https://github.com/AlvinRamoutar/tracert2csv

How to use

tracert2csv <PATH_TO_TRACERT_EXPORT> <OUTPUT_CSV>

Input

Tracing route to somenextdomain.com [34.102.136.180]
over a maximum of 30 hops:

  1     3 ms    15 ms     2 ms  hitronhub.home [192.168.0.1]
  2   265 ms    49 ms    17 ms  99.227.240.1
  3    26 ms    22 ms    14 ms  24.156.158.245
  4    16 ms    12 ms    13 ms  9019-cgw01.ym.rmgt.net.rogers.com [69.63.248.205]
  5    14 ms    18 ms    17 ms  209.148.235.18
  6    17 ms    13 ms    14 ms  72.14.209.126
  7     *        *        *     Request timed out.
  8    15 ms    12 ms    13 ms  180.136.102.34.bc.googleusercontent.com [34.102.136.180]

Trace complete.

Output

1,3,15,2,hitronhub.home [192.168.0.1]
2,265,49,17,99.227.240.1
3,26,22,14,24.156.158.245
4,16,12,13,9019-cgw01.ym.rmgt.net.rogers.com [69.63.248.205]
5,14,18,17,209.148.235.18
6,17,13,14,72.14.209.126
7,*,*,*,Request timed out.
8,15,12,13,180.136.102.34.bc.googleusercontent.com [34.102.136.180]

That’s it. I’m committing Sudoku.

Being stuck home for the past month due to Covid-19 has not only given me drive to learn new things, but has also driven me to the walls with boredom.

After all, what’s the use of learning something new if you don’t apply it?

And thus, sudoku_api was born.

sudoku_api is a pure dart package which simplifies the generation, interaction, and management of Sudoku Puzzles.

Well, how does it do that?

  • Contains a Built-in solver for generating new puzzles, or solving existing ones
  • Utilizes and exposes event streams for grid interactions (e.g. cell value changed), and puzzle states
  • Exposes a bunch of neat, easy to use models manipulating Cells, Positions, Grids, and even Patterns of a Sudoku puzzle

Whoah that sounds pretty cool – but what’s this ‘sue dough que’ you keep talking about?

Sudoku is a logic-based, combinatorial number-placement puzzle. The objective is to fill a 9×9 grid with digits so that each column, each row, and each of the nine 3×3 subgrids (segments) that compose the grid contain all of the digits from 1 to 9. The puzzle setter provides a partially completed grid, which for a well-posed puzzle has a single solution.

Wikipedia

Let’s see some sample code

import 'package:sudoku_api/Puzzle.dart';

void main() {
  
  PuzzleOptions puzzleOptions = new PuzzleOptions(patternName: "winter");
  
  Puzzle puzzle = new Puzzle(puzzleOptions);
  
    puzzle.generate().then((_) {
      print("=====================================");
      print("Your puzzle, fresh off the press:");
      print("-------------------------------------");
      printGrid(puzzle.board());
      print("=====================================");
      print("Give up? Here's your puzzle solution:");
      print("-------------------------------------");
      printGrid(puzzle.solvedBoard());
      print("=====================================");
    });
}

Inspiration

I’ve mentioned a few years ago that one of the first games I completed, 100%, was Brain Age Sudoku on the DSi. I really loved the simplicity and clean, non-bloated interface.

Flash forward a few years, and I still haven’t found an app that gives me the same feel.

My goal is to develop a highly customizable, clean, and easy to use Sudoku Toolkit in Flutter using this API – and perhaps encourage others to do the same.

A solid chunk of logic was re-used (namely, solving) from a previous project in college; Java Sudoku. The rest was an experiment in developing a dart package for the first time.

Project GraPi – Completed Kiosk Hardware (#06)

Creating a health dashboard by hacking intelligence into an AOpen (the ‘A’ for ancient) monitor; with metrics aggregated by Graphite and beautifully displayed with Grafana.

This marks the completion of hardware needed for the GraPi system – our AOpen monitor now functioning as a smart Kiosk.

Minor mishap; realized at the end that the USB/Ethernet Hub’s mounting orientation results in the Ethernet port to the RIGHT of the monitor rather than the left.
However, I’m planning to hide these cables within the rail, so once the Ethernet cable gets pulled behind the monitor, it shouldn’t be noticeable.

You might be able to notice from the images above, but we do have a working Graphite & Grafana setup running – something I’ve been working on alongside all this, and will be discussing in the future.
The metrics seen are an example, with data from WorldPing targeting a friend’s server in the states – the purpose being to illustrate a loaded Grafana dashboard.

From here on out, it’s all software.

In the next few posts, we’ll cover setup of Graphite and Grafana on Ubuntu Server, followed by setting up a Grafana dashboard kiosk on Raspbian Lite.

Project GraPi – Why so slow, Raspberry Pi? (#05)

Creating a health dashboard by hacking intelligence into an AOpen (the ‘A’ for ancient) monitor; with metrics aggregated by Graphite and beautifully displayed with Grafana.

I’d like to say that the build went on without a hitch, but I ran into two issues. Not project critical issues, but troublesome nonetheless.

Let’s talk about them, and how I’ve managed to successfully resolve some dastardly slow performance, and fail to resolve high static audio issues.

Troubleshooting poor processing performance

Super excited to get the system going, I dd’d Rasbian Buster (desktop) onto a 16 GB card, and completed setup for my Pi Zero.

Immediately after boot, I noticed how sluggish the system was. Some symptoms I’ve experienced include:

  • Desktop environment input lag (mouse movement, key presses)
  • Long time to load applications
  • SSH authentication takes abnormally long (reached 10 seconds at one point)
  • SCPing at a rate of 10 kb/s over Ethernet
  • CPU always at 100% utilization

Half of me thought it had something to do with my configuration.
However, these symptoms were prevalent since the first boot into Raspbian.
Furthermore, a majority of my configuration was network related, nothing that would cripple other system components.

The other half of me thought it had to do with my Pi Zero. I’ve done some reading online and found others also experiencing poor system performance with Raspbian Buster (desktop) on the Pi Zero. Perhaps a full desktop environment image of Raspbian was asking too much?

Well, I didn’t let that stop me, and took to exploring a wide breadth of light-weight OSes for the Pi, including PiCore, Raspup, and DietPi.

I’ve wiped this MicroSD card enough times to give it an identity crisis.

I eventually got tired of testing out different distributions, and defaulted over to Raspbian Buster Lite, where I installed LXDE along with Midori, Epiphany, and Chromium to decide which would serve as my kiosk browser.
The final image after setup served to be the most responsive but browsing a single page was still taking forever (>60 seconds to load alvinr.ca) – and even then, the load wasn’t complete.

The more numbers I put together, the stranger the situation looked.
It was here when I put software aside, and started looking at some hardware specifications

The Pi Zero runs a 1GHz single-core CPU with 512MB RAM.
My CPU utilization on a fresh installation of Raspbian Lite was always near 100%, while RAM usage on my own image never exceeded ~200 MB.

The entire unit remains quite cool while operating, barely deviating from room temperature – it wasn’t like I was giving it a reason to, nor the means to with a 5V/1A supply.

Huh.

Pi Power for Raspberry Pi Zero taken from FAQ

Taking a look at the power requirements for the Pi Zero made me realize I was using a supply below recommended. This was taking into account my ‘desktop’ setup, with a USB/Ethernet hub housing a wireless keyboard/mouse dongle.

Could it be? The Pi was throttling CPU to prevent brownout?

I’m telling you – if there’s anything I’ve learned as a FIRST alumni from 4 years of competitive FIRST Robotics, it’s to know your power draw, and always charge your damn batteries.

As for our system here, we know the power draw.
Some further research for the Raspberry Pi yielded the following documentation – recommending a supply of 2.5A.

I’ve got something close enough – a Samsung S7 5V/2A charger.
I swear, if I plug in this supply and it turns out that was the problem all along…

Well.


After an hour of playing around with it (and several sanity-driven reboots later), I now experience U N P A R A L L E L E D performance.
Really though, performance has improved drastically.

CPU utilization with Chromium running sits at around 60%, and it only takes 20 seconds to complete load alvinr.ca.
Network performance was better as well, managing to perform an SCP transfer at ~8 MBps – close enough to the 10/100 Ethernet adapter capabilities.

All that’s left now is to replace the existing 5V/1A adapter with this one.

Sweet – let’s move on.

Troubleshooting high static on the Raspberry Pi

It’s the end of the final assembly – we’ve got a better power supply, wires and components have been neatly organized, and everything is secured down.

Let’s perform one more test – except instead of loading alvinr.ca, let’s load SoundCloud and listen to a cover by Anson Seabra – Love The Way You Lie by Rihanna.

I couldn’t even make it 10 seconds in – the sound was absolutely jarring, a rough mix of tones and high static.

Audio recording of the playback of the first ~30 seconds of Anson Seabra’s ‘Love The Way You Lie’ cover

Not the song mind you – I’ve definitely broke something along the way.


Let’s get this out of the way – I haven’t resolved this yet, nor have I pinned down an exact cause. I’m moving forward with the project, considering all the time invested into troubleshooting (and since audio was an optional feature).

However, if you manage to run into something similar, here are some things I’ve tried:

  • Re-seat connectors
  • Use a shielded vs. non-shielded AUX cable
  • Ground Raspberry PI
  • Different audio output device
  • Different audio track, different audio player, even different Linux distros

At the least; I’ve narrowed it down to a hardware fault, either with the Pi, or more likely, the Mini HDMI to VGA adapter (since it splits audio to AUX output).

A little light and some hot glue digging eventually led me to finding a possible fault, present in this very adapter.

Small, purple wire disconnected from the main PCB of the Mini HDMI to VGA adapter

While I don’t know what purpose this wire serves, I’ll leave the blame on it for now, until I find the time to either investigate it further, or purchase another adapter.

Extending vDisk in FreeNAS VM

A journey in extending a ZFS volume pool on a FreeNAS virtual machine after changing vdisk size.

Skip to the procedure section to dig right into it

Background

Virtualizing I/O is perhaps one of the best ways to shoot yourself in the foot.
Or IS it?

You’re taking a piece of software of which its very namesake involves large I/O operations, and throwing it behind layers of abstraction.
When an I/O request is made from a VM, the hypervisor must both ascertain the source, and gain access to the destination by translating the abstracted storage location to a physical location – all while remaining secure.

Not to mention;

One does not simply host a hypervisor for a single VM

Your I/O heavy VM is contending for bandwidth across an already busy storage driver, to some storage which may also be shared by other VMs.

Furthermore, if you’re not even running a bare metal hypervisor and are using applications such as VMware Workstation or Oracle Virtualbox, then performance degrades further – since your hypervisor is now contending against the host operating system for I/O.

Why then, would one want to virtualize NAS?

As much of a sin this seems to be, there are some clear and powerful benefits to doing so.

Perhaps most obvious, you’re given the flexibility provided by any virtual system; simplified management, accurate hardware performance metrics, and one-click customization should you wish to change CPU/RAM/disks.

This ability to change the size of a VM’s disk is quite trivial.
Getting FreeNAS to realize the added space is another story altogether.

Furthermore, there are ways to eliminate I/O bottlenecks by using PCI pass through, giving VM access to raw disk.
Do note that certain hypervisor features are unavailable when doing so – taking VMware for example, where you will lose fault tolerance, HA, DRS, snapshots, and a few more.

And then there’s laziness factor; why go through the effort of setting up a whole new, dedicated machine just for NAS when I can just spin up a FreeNAS instance in minutes?

My use case is personal, and while certain pros/cons hold true between lab and production environments, research is key in determining whether NAS virtualization is best for you.

Procedure – Extending FreeNAS Pool Size after vDisk size change

When you increase the size of a virtual disk, it’s up to guest to resize/grow partitions to utilize the free space.
This is what the following procedure will cover; how to properly resize a volume after changing one of its disks’ dimensions – and having FreeNAS update pool size accordingly.
Note that the following procedure applies to FreeNAS 9.2.1 – however, it remains relevant for newer FreeNAS versions.

Step 1. Log into webmin, detach the volume being resized

Log into your FreeNAS web administration portal.
By detaching this volume, we can safely perform partition changes to disk(s) within.
UN-CHECK the destroy data and delete shares options.

Step 2. Shut down FreeNAS

Ensures that there are no lingering locks to this volume and/or its disks before we grow our drive(s).

Step 3. Grow drives

Grow vDisks using whatever means available – vboxmanage, vmkfstools, vmware-vdiskmanager, etc.

Step 4. Power on FreeNAS, and enter shell

Step 5. Enter the following commands in order:

zpool status

Retrieve the name(s) of partitions for each pool. One of these partitions will correlate to the disk(s) expanded.
NOTE down the gptid for YOUR pool – we will use this later.

glabel status

Resolves a partition name to drive name (Components column).
NOTE down the Components name for the partition identified previously – we will use this later.

gpart resize -i 2 /dev/da0

Replace da0 with drive name (Components name) identified previously.
This command will resize the specified disk.
On completion, a ‘resize successful’ message is given.

Step 6. Log into webmin, and auto import the pool

On successful auto import, you will see your volume under Storage > Active Volumes

Step 7. Bring device back online and expand pool size

Return back to your FreeNAS shell, and enter this final command to both bring the device online, and automatically update the pool size.

zpool online -e poolname /dev/gptid/some-long-gptid

Replace poolname with the component name from zpool status.
Replace some-long-gptid with the partition name from glabel status.

The ‘-e’ flag is meant to be used when a smaller disk has been replaced by a larger disk – which to the guest, is what we’ve done.

At this point, you should be able to refresh your webmin and note the changes in used/available/size for your particular volume in Storage > Active Volumes.
This of course, will also be reflected in any shares active.

Happy virtual NASing!


Further reading

Dingding Li, Hai Jin, Xiaofei Liao, Yu Zhang, Bingbing Zhou, Improving disk I/O performance in a virtualized system, Journal of Computer and System Sciences, Volume 79, Issue 2

Yes, You Can Virtualize FreeNAS, May 12, 2015, FreeNAS, FreeNAS Blog

Project GraPi – Loose ends (#04)

Creating a health dashboard by hacking intelligence into an AOpen (the ‘A’ for ancient) monitor; with metrics aggregated by Graphite and beautifully displayed with Grafana.

As promised, let’s dive into some of the modifications made to Audio Control and USB/Ethernet Hub – all part of the master plan.


Audio Control – Moving Audio IN, INside

With our cheap little Mini HDMI to VGA adapter providing audio output from the Pi, all we need to do is feed it into the monitor. Spoon of choice? AUX cable.

Now here’s where it gets a little awkward.
With the unit closed, Audio IN is facing outwards. Meaning we’d have to lead an AUX outside the unit and plug it into the back, making this odd protrusion.

Well, that won’t do.
One of the key tenets of this project was to ensure the unit is compact, with as many components encapsulated within the monitor chassis itself.

Let’s perform a surgery, and extract the Audio IN (AUX) port and see if we can move it around.

And done. Time to test.
Powered on the unit, ensured the AUX cable was connected between our Mini HDMI > VGA adapter and Audio IN on the monitor, and basked in the graininess of Do You Remember remixed by grey.

Yeah, it was pretty bad.
Not because of the mod I assure you (having tested audio on this monitor awhile back), more so because the speakers are terrible.

Anyways, won’t be playing music through this anytime soon – any audio coming through the Pi will be for notifications/alarms.


USB/Ethernet Hub – Mounting

Never thought I’d say it, but for once, this modification was made EASIER by the cheap construction of a part.

The Ethernet/Hub adapter easily came apart (surprise), separating away from its case, allowing me to determine a suitable mounting location.

Deciding to mount it near the bottom for both out-of-sight accessibility came with the added bonus of being closer to the Pi.
To mount the hub, I ran one of the leftover hinge screws through the lower-half of the plastic hub case, right back into the frame.

All that was left was to cut a portion of the rear cover out to accommodate the area now taken by the hub, and we were good to go.

And there we go!
This wraps up a majority of the hardware.
For now.

The next post will cover assembly in more detail, along with any issues we should consider before side-mounting onto my server rack.

Project GraPi – Cable Prep (#03)

Creating a health dashboard by hacking intelligence into an AOpen (the ‘A’ for ancient) monitor; with metrics aggregated by Graphite and beautifully displayed with Grafana.

When it comes to hardware, I’ll be keeping things simple.
With the exception of the monitor itself, the other complex subsystems are as follows:

  • Raspberry Pi Zero
  • USB to USB/Ethernet hub

The remaining components are internal wires/cables, which includes the Mini HDMI to VGA adapter, and 5V/1A wall adapter.

Internal Component/Wiring Diagram

GraPi Internal Component/Wiring Diagram

Legend for above diagram:
Bolded titles are components ADDED to the vanilla monitor system.
Italicized titles are components which had to be modified in some way.
Red lines are cables which had to be modified in some way.

Component boxes nested within other component boxes are either mounted, or exist, on that component.

Let’s talk modifications

Certain cable/component modifications were pretty simple. For instance, the Mini HDMI Cable was folded at one end and held as such via some electrical tape. Other simple modifications included the Generic USB Cable used between the USB/Ethernet Hub and Pi Zero, and the Generic AUX Cable – both of which were looped and tied.

The rest were a little more complex; take the 5V/1A wall adapter for instance.
I didn’t bother hacking it open because the case played a valuable role in its mounting point.
Therefore to supply AC, I split the input from the Power Distribution Unit to both the unit itself and the 5V/1A wall adapter, connecting directly to the non-polarized plugs and wrapping the connection in shrink wrap.

USB to USB Micro B cable

The USB to USB Micro B cable was drastically shortened to 30cm, and rejoined via small solder joint, and shrink wrapped.

The HDMI to VGA adapter was stripped of its hideously large housing and plugged into the monitor’s VGA cable.

At this point, wiring video would be complete in a traditional monitor. However, this monitor has its VGA cable BUILT-IN. Man, was this an annoying realization.
Fortunately, the other end wasn’t soldered onto the main board, and was rather some rather delicate AOpen proprietary connector.

Messy SVGA cable splice (final cable after cleanup looked much better, trust me)

To ensure the integrity of this connector, I ended up splitting and shortening the built-in VGA cable to 30cm as well.

Cut SVGA Cable from SuperUser

If you’ve ever wondered how all those pins in a VGA interface are sent along a single cable, I highly suggest you split a cable open to learn how.
It’s an absolute beast of a cable, this particular one having a woven metal sheathe surrounding 3 coaxial-like cables and 9 wires.

Soldering individual strands was challenging enough with the variety of stranded conductors. Keeping stripped conductors from contacting was a whole new bag of worms.
Hot glue came in useful for once, and helped not only hold the splice together, but keep each individual wire in place, preventing contact.

Cables connected, fingers crossed, let’s power it on

GraPi Monitor, first power on test
GraPi monitor, first power-on test with core components

IT’S ALIVE!

Thank the stars that this picture excludes the mess behind the monitor 😉
This covers all the core system components.
We’ll be wrapping up on hardware soon – man, I can’t wait to put the rear cover back on.

We still need to talk about the USB/Ethernet Hub, and the Audio Control (both of which requiring some modifications), but we’ll leave that to the 4th GraPi post.

Project GraPi – Boom, CLAP the sound of my Goals (#02)

Creating a health dashboard by hacking intelligence into an AOpen (the ‘A’ for ancient) monitor; with metrics aggregated by Graphite and beautifully displayed with Grafana.

Let’s get some simple objectives and materials down before I change my mind.
That way, I’ll have a post to update when I do.

Requirements

  • All hardware must be contained within the AOpen monitor.
  • Monitor should remain mountable.
  • Minimal power consumption from new hardware.
  • Minimal heat dissipation from new hardware.
  • Hardware (compute) should be sufficient to load a live Grafana dashboard.
  • Audio playback though integrated speakers for alarms (optional).
  • Switchable to secondary input (optional).

Equipment

* no guarantees that any of these links will remain functional

Project GraPi – Inception (#01)

Creating a health dashboard by hacking intelligence into an AOpen (the ‘A’ for ancient) monitor; with metrics aggregated by Graphite and beautifully displayed with Grafana.

Background

Killing Floor 2 can be pretty fun, especially with a great crew. Not to mention the amazing community, which continues to pump out workshop content such as custom maps. With a crew assembled from players around the world, we casually run games now and then, a solid amount on custom maps.

Well, one of the servers I manage went down for some time.
It wasn’t until further investigation that I discovered the host ran out of disk space with all the custom maps we’ve loaded.

It was a minor inconvenience that resulted in some down time. However, it was something that would’ve easily been foreseeable, assuming I was monitoring the host.

This made me realize something; a pretty solid chunk of services I host for both external and internal use are manually monitored.

And by that, I mean I remote into them whenever the fit hits the shan.

Nowadays, tools which help in application monitoring and insight are bountiful. It would’ve been easy enough to just pick one and deploy.

But that would be boring.

So, in combination with the need for some app. monitoring & insights, let’s build our own monitor (literally), which will be mounted on my server rack with the sole purpose of displaying a comprehensive health dashboard.

So, where we headin’?

Over the course of the next few weeks, I’ll be further covering specific goals, documenting the progress already made, ranting on my failures, and generally discussing what I’ve learned – and will learn along the way.

To make following along simple, each post dedicated to this project will not only be categorized grapi, but also have its titled numbered in order of chronology and relevance.