16Mar/20

Coronavirus Dashboard – covid19.py

To view the covid19.py dashboard click here: covid19-normal.html

To view the log scale covid19.py dashboard click here: covid19-log.html

Source code (nothing fancy; it uses plotly for the charts): https://github.com/bhbmaster/covid19 . This dashboard gets its data from https://pomber.github.io/covid19/timeseries.json

The charts are updated every 6 hours starting at midnight PST. However, the values to the data source are updated daily, so don’t expect new values until after midnight.

17Jan/20

Suggested Robocopy Switches

Note: This article is not mine. It was written on http://www.rainingforks.com/blog/2015/suggested-robocopy-switches-explained.html. I am simply excerpting the whole article. This way if that site ever gets shutdown, at least there will be a copy here. I am not taking any credit for the material on this post. This article was written by Steve Schuler.

Suggested Robocopy Switches Explained

Windows’ robocopy.exe is a great command line program to quickly copy or fully backup your files, but there’s a lot of confusion out there about how to use its (not very well-documented) switches.  Here’s just what you need to know:

First of all, you probably already know that typing “robocopy /?” will give you a long list of switches to choose from.  Start there if you’re confused.  But since there are a LOT of choices, and they’re not well-explained, here’s a run-down of what I typically use, as I go about my day as an IT guy:

The basic format is: robocopy <source path> <destination path> <switches>

NOTE: I’ve found that using robocopy to copy across a network doesn’t always work using mapped drives!  Instead, use the full path (especially important when running as a Task in Windows Task Manager). For example, instead of “robocopy C:\Foo S:\Foo” do this: “robocopy C:\Foo \\SERVER\Foo

Another tip is if you’re using file paths that contain spaces, then you need to enclose each path in quotes. If no spaces, then quotes are optional.

/FFT is necessary to copy between file systems, such as Windows’ NTFS and Linux’s EXT4. If you don’t use this you can get wierdness like files looking like they’re newer than they really are, etc., since the two file systems keep time differently.

/COPYALL copies ALL aspects of the file/directory, including ownership and permissions info. Required if you’re backing up a server or something that you want to maintain group/user permissions, etc. for. (NOTE: Don’t use this switch when copying files from Linux to Windows if you aren’t logged in as the same user with admin rights on both machines! If you do, you’ll get a lot of errors like “A required privilege is not held by the client” and “The revision level is unknown” as it creates a bunch of empty folders at your Windows destination, but skips copying all your files! Instead you can use the /COPY:DT mentioned below, and if you really need to backup your Linux ownership & permissions info, save all the files in a tarball and then just backup that single file containing the directories & files with their attributes intact to Windows.)

/COPY:DT to just copy files & date/time stamps. This is good if you’re just copying some files to give to a friend, and don’t need permissions, etc. copied. Also good for copying files from Linux to Windows (see “note” in /COPYALL above).

/FP outputs the full path so you can more easily see where it is while it’s running.

/MIR exactly mirrors the files & directories, so things at the destination will be deleted if they’re not at the source. (This is the same as using /PURGE (which deletes stuff at the destination that doesn’t match) with /E (which includes Empty subdirectories)

/ZB tells robocopy to use restartable mode (which you want for large files, especially over WAN/unstable connections, since it’ll try to pick up where it left off if the connection gets dropped or there’s corruption mid-copy), and if access is denied, then it’ll use Backup mode, which allows you to copy files you might otherwise not have access to, assuming it’s being run under an account with sufficient privileges (e.g., member of Backup Operators, Administrators, etc.). (NOTE: the /Z switch sometimes slows down the copy speed, so if you don’t need it, don’t use it, especially if you feel like your Robocopy job is taking longer than it should. Sometimes there’s no speed difference, and sometimes it can be dramatic.)

/MT stands for Multi-Threaded, and tells robocopy to copy multiple files at once. The default number of threads is 8 (max is 128), but be careful, as running this over a network can really saturate your bandwidth, leaving none for anyone else. As a result, you may want to skip this one or try specifying less threads by doing something like /MT:2 which will just run two threads (instead of 1, which is what you get if you omit /MT entirely). (NOTE: This is only available in newer versions of Robocopy (Win7/2008R2 and later).  If you’re running older versions (or just don’t feel like bothering with this switch), you can simply open multiple command prompt windows and run it in multiple instances – I often will run two or three Robocopy batch files simultaneously. Also, this switch will make file copy progress numbers confusing, so it’s best to use the /NP switch mentioned below to disable outputting the copy’s progress. Some people speculate that running multiple threads can increase fragmentation, but I haven’t seen any hard evidence of this, and with increasing adoption of solid state drives, it may not matter for much longer anyway.)

/R:1 /W:3 are two switches you probably want to use together to tell robocopy how many times to retry accessing a file (1 in this example), and how long to wait between retries (3 seconds in this example).  If you leave this out, it’ll retry 1 million times with a 30 second wait between each one when it encounters a file it can’t access!!!

/XD is what you use when you want to tell robocopy to skip (i.e., exclude) a directory. Just follow /XD with a space and then the path to what you want excluded. If there are multiple directories you’d like to skip, separate them with a space. For example: /XD “C:\Foo\private stuff” “C:\Foo\plans for world domination”

/LOG:C:\LogFileName.txt /TEE /NP are three switches you’ll want to use together if you want to write the results of the copy to a log file (called “C:\LogFIleName.txt” in this example). If you want it to write what’s happening to the screen as well as to the log file, then you’ll also want to include /TEE.  And, possibly most importantly, you want to include /NP in there so that it does NOT show the progress as each file copies. If you leave this out, then your log file will be filled with every single percentage complete it displays! So you’ll have something like this: “0.0% 0.1% 0.2%” and so on, to 100% FOR EACH FILE, which is nuts.

Finally, if you want to append log file output to the end of an existing file, rather than creating a new file every time, you can modify the above line to include a plus sign, like so: /LOG+:C:\LogFileName.txt

That’s pretty much all you need to know! I figured most of this out from trial & error, as well as some internet searches. Unfortunately, I’ve read so many incorrect, or confused posts about robocopy on the internet over the years, that I wanted to write this to set things straight.  Hopefully you find this helpful – if nothing else, at least I’ll now have a handy reference for the next time I need to put together a robocopy command… 😉

24Dec/19

How to RDP over an SSH tunnel

You can use this trick to access your home PC from a remote PC (like your work PC). Instead of using Teamviewer or other similar software. You can setup up your own encrypted and secure tunnel to work thru. The requirements are basically to have an SSH accessible server in the same network as the PC you want to access.

Server Side Requirements:

  1. Windows PC that you want to connect to (example local IP: 192.168.1.3)
  2. Linux server with SSH in the same network as the Windows PC (example local IP: 192.168.1.2 over port 22)
    1. This can be a virtual machine running off the Windows PC (just as long as it is accessible from the router; so make sure you use a Bridged Network Adapter)
  3. Internet access to the Linux server SSH (example WAN IP: 1.1.1.1)
  4. This can be achieved by setting up a port forward on your router to send traffic destined to port 22 (or any port) from your Router to the Linux server’s port 22.
    1. Example1: route traffic hitting 1.1.1.1 on TCP port 22 to internal port TCP 22 on 192.168.1.2 (example: we port forwarded port TCP 22 from the router to 22 on the linux server)
    2. Example2: route traffic hitting 1.1.1.1 on TCP port 12345 to internal port TCP 22 on 192.168.1.2
  5. Enable RDP on your Windows PC:
    1. Control Panel -> System and Security -> System -> Change Settings -> Remote -> allow RDP connections && uncheck the box “Allow Connections only from computers running Remote Desktop with Network Level Authentication (recommended) -> Select Users and add the Windows User[s] that will be connecting to the RDP

Client requirements

  1. For Windows machines connecting to the rdp tunnel: Make sure your Windows client has Cygwin installed with ssh program (The windows client is the one used to connect)

Verification

  • Verify the setup works by SSHing to your Linux server from a remote location.
  • Also if you can try to connect to your RDP from another PC in your home network. Windows+R then type “mstsc /v:192.168.1.3:3389

How to connect from a Window PC:

For the sake of the example I will use the IPs highlighted as examples.

Open cygwin and run “./sshrdp_cygwin.sh 192.168.1.3:3389 root 1.1.1.1 22” then put in your SSH password. Then the RDP window opens and put in your Windows Login credentials

You can then make an alias in your ~/.bashrc script to alway connect to your homepc (assuming you put sshrdp.sh into your /usr/bin directory).

alias homepc=’/usr/bin/sshrdp_cygwin.sh 192.168.1.3:3389 root 1.1.1.1 22′

From then you can just type homepc on your cygwin and it will launch up

Connect from a Mac

Follow the same steps as Windows but use this script instead. Also since MACs don’t have mstsc. You will need to install and open up RDP software manually each time a tunnel is setup. The sshrdp script will prompt for your SSH server address & give you instructions like

“Open RDP to localhost:10000”

Then you will need to login with your Windows credentials

Here is the MAC version of the same script:

Similarly, you can setup an alias to use on your MAC terminal, except you will need to put it in your ~/.bash_profile instead of your ~/.bash_rc (if I recall correctly, that is how it is done with MACs)

alias homepc=’/usr/bin/sshrdp_mac.sh 192.168.1.3:3389 root 1.1.1.1 22′

Then you can access your home pc by simply typing homepc.

Connect from a Linux Server

You probably just use the MAC steps – although I am not sure and have not tested it. The line of code with the netstat command might need a change / edit.

18Jul/19

Bash convert seconds to human readable

function displaytime {
local T=$1
local D=$((T/60/60/24))
local H=$((T/60/60%24))
local M=$((T/60%60))
local S=$((T%60))
(( $D > 0 )) && printf ‘%d days ‘ $D
(( $H > 0 )) && printf ‘%d hours ‘ $H
(( $M > 0 )) && printf ‘%d minutes ‘ $M
(( $S > 0 )) && { (( $D > 0 || $H > 0 || $M > 0 )) && printf ‘and ‘
printf ‘%d seconds\n’ $S; } || printf ‘\n’
}

-bash-4.1$ displaytime 604800
7 days
-bash-4.1$ displaytime 604801
7 days and 1 seconds
-bash-4.1$ displaytime 800
13 minutes and 20 seconds
-bash-4.1$ displaytime 780
13 minutes

28Apr/19

Wrong Port Connected Check With ifconfig

We need to run ifconfig and look for the status line.

Remember this about ifconfig output:

* if “status: active“, the port is linked up and sending traffic (now you just need to make sure it has an IP for it to be able to communicate on a network). You can see it is sending traffic with tcpdump -i <port>
(hit Control-C after a few seconds; even ports that are linked up w/ a missing IP should still be sending out and receiving some L2 switch traffic: STP, ARP, LLDP)

* if “status: no carrier“, the port is not linked up and therefore not sending traffic . If you run tcpdump on this port, you will see 0 packets.

* if “status:” line is missing, then the port is disabled. Enable it w/ “ifconfig <port> up” to see if it has a link (you can optionally disable the port afterwards if you see it has no carrier using “ifconfig <port> down“)

18Apr/19

Grepping Thru All Rotated Logs

If your system gzips and rotates your logs.

You can use this trick to search thru all of them.

# zgrep -i — “SEARCHTERM” $(ls -1Str /var/log/messages*)

Another way

# zgrep -i — . $(ls -1Str /var/log/messages*) | grep “SEARCHTERM”

Sidenote: if you get an error try to replace “zgrep” with “zegrep”

03Apr/19

A good free syslog server

Personally, I just tried “Syslog Watcher” version 5.0.4 and it does a good job. I can start it by hitting “Start Server” then configured syslog clients to point at it. You can click on a message, then it will show up in the message view with every detail. Additionally you can hold Control and Mouse Wheel Down or Up to zoom in or out (just like in Notepad++)

Of course everyone is aware of Kiwi Syslog Server. I find Syslog Watcher more interesting, but that is just me.

27Apr/18

Find UPNP devices on your network – ex: ReadyDLNA or UPnP routers

Got info from here: https://stackoverflow.com/questions/18363833/how-can-i-list-upnp-server-renderer-in-command-line-console-mode-on-linux

Get the Upnp test tools

Example 1:

Find all Upnp devices (note their target names start with urn:…. It will hint at what service it is, ex: Layer3Forwarder, MediaServer, etc..)

Example 2:

Find DLNA servers which are served via Upnp (which has a target of MediaServer, urn:schemas-upnp-org:device:MediaServer:1)

The end

24Jan/18

bash simple variable substitution

You can use bash to do simple substitutions of variables, just like you can with sed. The bash trick turns out to take up less characters (bytes).

Lets set some variable

To replay it back:

or

Output of both:

 

Format

To do a simple substitution of the value/contents of a variable structure it like so. Put in the from part the thing you want to change (it can be a char or some chars), then it will replace them with the part you put in to (which can be a char or several chars). Note if mentioning special chars to escape them (example \ needs to be \\). Spaces are not considered special chars in this case.

echo ${VAR//from/to}

Then all of the parts that are from will change to to when the variable VAR is called.

Example 1

Lets replace all s with a S. You have to use the second notation type and add some parameters

Output:

  • Note: sed can be used to do the same substitutions

or

 

Example 2

Let replace all is with IZ

Output:

  • Note: sed can be used to do the same substitutions

or

 

Example 3 – filenames with spaces

What about converting an absolute path which has spaces to something with escaped spaces (backslash followed by space)

Output:

We convert spaces ” ” to “\ ” however in bash to print a backslash “\” we need to put an extra one “\\”

Output:

  • Note: sed can be used to do the same substitutions

or

The end.