Twice as (n)ISCE

UPDATE (August 2020):

In honour of ISCE release 2.4, with the removal of the distasteful terminology that cluttered previous editions, I have been trying to install it on my Windows 10 PC (with Windows Subsystem for Linux 2, running Ubuntu 20.04).

I managed to get a clean clone of the isce2 repo to compile, but not without incident. I had to hack around with several files to get it to work.

Here is what I did...

Get the repo, link the directory:

git clone https://github.com/isce-framework/isce2

ln -s isce2 isce 

Install dependencies:

sudo apt install scons libfftw3-dev libxm4 libmotif-dev python3-dev python3-pip snaphu cython3 libgdal-dev python3-gdal libhdf5-dev opencv-python libopencv-dev

sudo apt install scipy numpy h5py

I had to edit some files (paths from the isce2 directory):

1) contrib/geo_autoRIFT/autoRIFT/bindings/autoriftcoremodule.cpp - insert 

#include "opencv2/imgproc/types_c.h"

after the other include commands

2) components/zerodop/SConscript - commented out statements from line 36 to end

3) PyCuAmpcor/SConscript - commented out statements from line 14 to end

4) test/SConscript - commented out statements from line 23 to end

2) and 3) caused the compilation to crash, as I was not compiling with CUDA. You probably need them if you are. 

My new .isce/SConfigISCE : 

PRJ_SCONS_BUILD = /home/gareth/pkgs/isce/build

PRJ_SCONS_INSTALL = /home/gareth/pkgs/isce/isce

LIBPATH = /usr/lib /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/hdf5/serial /usr/local/lib /home/gareth/pkgs/isce/lib /home/gareth/pkgs/isce/lib64 /home/gareth/pkgs/isce/lib32 

CPPPATH = /usr/include/python3.8 /usr/include/python3.8/numpy /usr/include/hdf5/serial /usr/include/gdal /usr/include/opencv4 

FORTRANPATH =  /usr/include

FORTRAN = /usr/bin/gfortran

CC = /usr/bin/gcc

CXX = /usr/bin/g++

MOTIFLIBPATH = /usr/lib/x86_64-linux-gnu

X11LIBPATH = /usr/lib/x86_64-linux-gnu

MOTIFINCPATH = /usr/include  

X11INCPATH = /usr/include  

(substitute 'gareth' for your own username!)

My ISCE_config.sh script is now:

export ISCE_INSTALL_ROOT=/home/gareth/pkgs/isce

export PYTHONPATH=$ISCE_INSTALL_ROOT:$PYTHONPATH

export ISCE_HOME=$ISCE_INSTALL_ROOT/isce

export PATH=$ISCE_HOME/applications:$PATH  

(substitute /home/gareth/pkgs for wherever you cloned the github repo)

I added the following lines to the end of .bashrc to tell the shell about my install:

export SCONS_CONFIG_DIR=/home/gareth/.isce
source /home/gareth/pkgs/isce/ISCE_config.sh

(again, edit to point these to your own directories and script, whatever they are called)

When compiling, scons wasn't finding the dependencies, so 

scons install --config=force

gave it a bit more encouragement. And it ran to the end.

Good luck if you're trying it yourself! I'm off to provide 'feedback' on the process to the developers...

Hopes no longer dashed in Windows 10

I have largely weaned myself off of my old Mac laptop (a 2010 MacBook Pro) in favour of my newer, spiffier Microsoft Surface Pro 3. The SP3 is slim, weighs less than a kilo, which makes it a dream to carry about, and has functionality that a Mac can only dream of (a touchscreen! a stylus!) The Mac is now retired to sofa duty in my living room, while the SP3 is my new world travelling companion.

But not everything is automatically better. At least, not right away. There are a few little things that were so easy to do on the Mac that are nigh-on impossible in the Windows 10 operating system on the SP3. And the one that really bugs me is a little bit grammar-nerdy: en-dashes.

As an undergrad, I used to work as a sub-editor on the university newspaper. In essence, my job was to proofread chunks of the copy that eventually made it onto the pages. And one of the most common corrections I had to make was switching hyphens - these weedy things - for en-dashes. The en-dash is a dash the width of an 'n' – a far more satisfying size – and is just the thing to add emphasis or clarification to a sentence.

On a Mac, you could get an en-dash by typing <option><hyphen>, but there is no easy equivalent for Windows. Within certain applications, like Word, you can set macros to autocorrect to dashes, but that doesn't exactly help with typing in a web browser (say). Some websites suggest that you can get there with Alt-codes, but the big catch there is that you need a full-size keyboard with numeric keypad to get those to work. Which the SP3 is notably lacking in. Well, today, I spent some time googling the problem, and was heartened to find that a) I was not the only person obsessing over dashes, and b) at least one of those people had figured out a fix:

AutoHotKey is the simple solution to this problem. It's a free app that lets you set keystroke combinations to perform any action in any application, including browsers. I was able to map en and em dashes to alt-hyphen and shift-alt-hyphen, respectively. Works flawlessly — every time. (See what I did there?)

You'll see some discussion about this not working with Windows 10, but that's old discussion. It's working for me on my Surface Pro 4 just dandily as of January 21, 2016.

Here's what you do:

1. Download and run AutoHotKey (autohotkey.com)

2. Right-click on your desktop, and choose New > AutoHotKey Script. Name the script with a .ahk at the end.

3. Right-click the script, and select Edit Script.

4. Paste this code in that script below the existing text:

!-::–
return

+!-::—
return

5. Save, close and double-click the script to run it. Test it out in some random program. You'll see it now works.

6. Make a shortcut of the script, and move it to your Startup Items folder. (Windows-R, then type "shell:startup" to open the Startup folder.) Drag your shortcut into it.

I followed the instructions, and can confirm that they work a treat. Now to write some dashing prose!

Hat tip: David R Nagel on this thread.

Giving Linux in Windows a bash – Part 2: How-to

So you want to tool up your Windows 10 install to include the Linux bash shell, eh? Have dreams of natively running compiled Fortran programs in a terminal window on the same PC that you use to craft your PowerPoint slides? Want to make GMT figures and then edit them immediately in Adobe Illustrator? (You say yes and yes? Are you me?) Well, I've started to figure these things out, and I'll share with you how I did it...

[Almost all of this could be cribbed from other places on the web, if you can be bothered; this is blog post is more of a compilation. If you want to track things back to the original sources, this is where I found instructions for changing to Insider Previews; I used these suggestions to try and get Windows Update to work with the previews; I downloaded a working ISO file for a suitable update here; this Microsoft blog showed me how to activate the bash shell; I found out about configuring the X server here; and there was also a subreddit with some useful background.]

1. Update to a suitable build of Windows 10

The bash shell interface is set to be included in the Windows 10 Anniversary Update, which is rumoured to be released in July. Since it isn't out yet, you will have to update to a 'preview' (read, beta) build of Windows to do that. All builds after version 14316 have bash included as an option, so go for one of those.

• To check your operating system version, you need to take a short trip into the settings:
• Click on the start button and then on the 'Settings' icon.
• In the window that pops up, click on the 'System' icon.
• From the sidebar menu that appears, scroll down and click on 'About'.
• On the screen that appears, look for 'OS Build' and the number that goes with it. If it is above 14316, you don't need to do anything more here. 

  

  The Insider Program website. Yesterday.

• If it turns out you do need to install a newer build of Windows, you first need to join the 'Windows Insider Program'. This is associated with your Microsoft account, and so you'll need to log in to Windows using it from here on in, if you aren't already.
• Go to the Insider Program website, (pictured, right) and answer the questions there. It's free to join. 
• Make sure you're logged into Windows with your Microsoft account.
• The next step is to enable Insider Previews inside Windows. This requires some changes to your settings. 
• Click on the start button and then on the 'Settings' icon.
• In the window that pops up, click on the 'Update and Security' icon.
• From the sidebar menu that appears, click on 'Windows Update'.
• Then click on 'Advanced Options'.
• Scroll down on the screen that appears, and click on the button that says 'Get Started' under the heading 'Get Insider Builds'.
• A slider will appear with three positions, representing different levels of involvement. I chose 'Slow' – this means you only get preview builds that have been reasonably stable under testing. (If you like to live life on the edge, and can handle the possibility of the occasional less-vetted, unstable update, choose 'Fast'.) After a quick restart of your computer, you should now be ready for updates... 

In my case, having followed all these steps and restarted, I checked Windows Update, hoping to see details of my new version, and... nothing. Some searching of the Internets suggested that in some cases (but there was no firm sense for which cases), Windows needs a little prompting to turn up those updates:

• It may be that your Windows registry needs some 'adjusting'. This site has some suggestions about various hacks you can try. I tried all of them, including the suggestions found in the comments. They didn't seem to do that much good for me, but your mileage may vary.

In the end, I decided to update Windows manually using an ISO file (DVD image) of one of the preview builds: 

• Download a preview build from Microsoft's website. (The link is for build 14342, which is the one I tried.) You need to have registered for the Insider Program for it to work, I think.
• You get a choice of version, from a drop-down menu. Unless you have special circumstances, the top option ('Windows 10 Insider Preview - Build 14342') is probably the one to go for. Choose a language, too, and download the ISO file (should be 3.9 Gb or so).
• Once downloaded, double click on the ISO file to mount it.
• From the folder that appears, double click on 'setup.exe'.
• Follow the prompts to install the preview build on your computer. (There will be warnings... be brave and accept them.) You will have the option to keep your files and settings – you'll probably want to do that. I found the install went much faster when I told it not to apply any updates as it went along, but your experience may be different.
• After a series of restarts and logging in, you will eventually find yourself being introduced to a new version of Windows 10. Success!
• Windows Update should now (hopefully) show you another update that can be installed. Mine showed an update to Build 14366, for instance. You'll probably want to do that – everything that I show you below worked better with the new update than it did before it...

2. Activate and install the bash terminal

The Ubuntu bash terminal, the thing we want to install, goes by the rather unsexy name of the 'Windows Subsytem for Linux (Beta)', WSL for short. You'll need to make an excursion into the deeper recesses of the Windows Settings and Control Panel in order to install it:

• First, turn on Developer Mode. 
• Click on Start, then Settings, then Update and Security, then For Developers.
• On the screen that comes up, click on the radio button for 'Developer Mode'.
• Next, install WSL.
• In the Cortana 'Ask me anything' text box at the bottom of the screen, type 'features'.
• From the menu that pops up, click on 'Turn Windows features on or off'.
• Scroll down in the window that pops up, and click in the tick box for 'Windows Subsytem for Linux (Beta)'. Then select OK. The package should download and install, and you will be prompted to reboot. Do.
• Next, run it for the first time.
• Type 'bash' in the Cortana 'Ask me anything' box, and press return.
• This will open a terminal window, which will ask you some simple questions. Yes, you do want to install it. It'll download more things, assuming you said 'y'.
• At the end, it should ask you to set up your Linux username and password. Do so.
• And then run it again!
• Now you should find 'Bash on Ubuntu on Windows' in the start menu, or through the Cortana text box (try 'bash' again). And it should work. Huzzah!

3. Get X-Windows working

It's all very well to have the bash shell working. But the beauty of Linux, Unix and friends is that command line programs can open a wide variety of graphical outputs either from your own machine or on remote machines, through the X-Windows system. And so, until you have that functionality, it will only be a pale shadow of what is possible. Luckily, it isn't that hard to sort out.

• Download and install the free VcXsrv X-Windows server application. (Internet people seem to think this is the best free option.)
• Run VcXsrv. You should see an 'X' icon appear in your system tray.
• Now set up bash to use your X-Windows server:
• In your bash window type:  export DISPLAY=:0 (no misspellings, please)
• You will need to install some apps that make X output to test it. Luckily bash comes with the Ubuntu/Debian package installer that does all the heavy lifting for you:
• sudo apt-get install x11-apps  will install a set of simple apps that use the x11 system (you may need to enter your password to start the install)
• xeyes  will generate a pair of graphical eyes that will follow your cursor around (sort of, it can be slow to update).
• You could also try xcalc or  xclock (wonder what those do?)
• To make the change permanent, you need to add the set-up information to your bash configuration file (.bashrc, in your home directory). If you don't know how to do that, try typing the following:
• cd
• nano .bashrc
• scroll down to the bottom of the file
• paste the  export command from above
• ctrl–x to exit, 'y' to save changes.
• You are now the proud owner of a (somewhat) functional Linux X-Windows system running natively on your Windows PC!

4. Things to try/figure out, and some observations

• I successfully installed GMT5 under bash by following the standard GMT installation instructions for Ubuntu. (This did seem to need the extra, post-install Windows Update mentioned above, however.) I haven't yet tried all the commands, but plotting a simple pscoast world map worked just fine.
• I also successfully installed gedit, the GNU text editor. It started out by crashing with every file save, but following the extra Windows Update (to build 14366) it seems much better behaved and less temperamental. I have been unable to change the configuration of the editor yet, though.
• I had trouble getting ghostscript to run initially, but this too seems to have been sorted out with the update. ghostview worked from the off, on the other hand.
• I have not yet located where the computer has situated the files involved on the windows file system. To be honest, I just haven't looked too hard yet.
• I do know, however, that /mnt/c is the location of the C drive on my PC in the bash terminal.

Giving Linux in Windows a bash – Part 1: Why bother?

For over a decade, Apple Macs have been the computer of choice for many of my professional colleagues in geophysics. Although there is an element of tribalism in the choice for some (especially among the Californian lifestyle set), and perhaps appreciation for the pretty hardware design carries rather more weight than it ought to for others, I suspect neither is the main consideration. For many, it's all about the operating system.

Mac OS X. Yesterday.

Mac OS X, launched in 2001, was revolutionary in that its back end (the guts of the operating system, behind the scenes) was based on BSD, a variant of Unix, the operating system beloved of scientists, sysops and servers in the 90s and earlier. (My undergraduate research, Masters research and early PhD research – all in geophysics – were all conducted on Sun workstations running a version of Unix.)  And Apple provided a terminal app, a native C compiler and an implementation of X-Windows, making people in the know very excited indeed. 

Suddenly, full-on techy types could do all of their programming, testing and running of their command-line based programs on a machine that could also run Microsoft Word and Powerpoint, and Adobe Illustrator! That's what sold me, back in 2003, when I bought my first Mac, a 12-inch aluminium PowerBook. And again several more times –when I started my faculty job, when my students needed new machines last year. As we approach the 12th version of OS X, Macs remain the best all-round machines for my work. But the advantage is not as clear as it might once have been.

Why, you might ask? Apple in recent years have not been the powerhouses of development and fancy hardware that they were in the past. The pro-level machines have become expensive, essentially unexpandable dustbins, with neither the aesthetics or upgrade path of my cherished 2010 Mac Pro desktop. The laptops have stagnated, with touchscreen technology passing them by, and the latest MacBook line is ridiculously undersupplied with ports (but overprovisioned with scandalously expensive dongles). Really, now, it's getting to the point where the obvious flaws in the hardware are starting to overwhelm the benefits of the software.

So, when my opportunity to get a new laptop through work came along, I bought myself a Surface Pro 3 which, minor foibles aside, has been a great purchase – light, flexible, speedy, with reasonable battery life and a stylus! I even managed to get it to dual boot, so that I can do proper work when travelling (although having to reboot it to run PowerPoint is annoying). And when I had money to spend on a desktop machine, I spent $2000 on PC components and built myself a awesome Linux-toting thunderbastard of a machine that can run all my research codes at express speed for a fraction of the price of a lesser-specced Mac Pro dustbin. If it could run professional application software (rather than the fairly weak sauce Linux alternatives), I wouldn't need anything else.

Which is why Microsoft's announcement that they are teaming with Canonical to produce an Ubuntu style bash terminal in Windows 10 is so exciting. It raises the possibility of doing everything on one machine again, but that machine could be a Windows machine. The tool is actually in beta right now, and if you enable Insider Previews on your Windows 10 install, you could have it today! I'll be giving that a go to see if it really is a solution to the 'one machine' problem... watch this space. 

Fun with StaMPS and Doris

Stamps, yesterday. (Picture from blogandcollectibles.com)

Although the title may make it sound like I am indulging in philately with an elderly neighbour, this is actually high-tech stuff. For those who are down with the lingo, this is the versatile and freely-available multi-temporal InSAR processing package StaMPS, written by the inestimable Andy Hooper of the University of Leeds. [Doris, on the other hand, is an inscrutable interferogram processor from the Netherlands that I have trouble understanding.] For those who are not down with the lingo, move swiftly along.

With the end of the teaching quarter (wahoo!) I find myself with a little time to indulge in doing science, involving actually looking at some data. [Of course, it shouldn't feel like an indulgence – it is, after all, my job – but after ten weeks of teaching, it seems like a real treat!] And, despite being a long-standing user of the code (one of the original users in fact), I am a little rusty. And so, for posterity's sake, I have documented here the steps it took to get me to where it might actually produce something... [update, 8/6/2016: Forgot to post this at the time, and now my memory of what I actually did is a little hazy. So the second half of this is a little more brief than you might like, blame the me from sometime last summer.]

tl;dr

If you are running StaMPS on a Mac, the FreeBSD version of the 'date' command is sufficiently different that it breaks the script that parses the timing information from the ROI_PAC metadata and writes it out in Doris format. And that messes up everything. See item 11 below for the fix.

Dorises, yesterday. (From Google Image Search)

1) The setup

The hardware: My trusty (but aging) 'cheesegrater' 12-core Mac Pro running OSX 10.10, with 64Gb of RAM and just enough disk space. Of course, the code was developed with the Linux operating system in mind...

The software: StaMPS v.3.3b1 (freshly installed) and Doris 4.02. ROI_PAC 3.0.1. Libraries installed via fink. MATLAB R2014b.

The data: 20 ALOS PALSAR scenes from New Zealand, from 2007–2011. (I have never processed ALOS data before in StaMPS...) A rather excessive SRTM DEM of the whole of South Island with holes filled in by the CGIAR consortium.

2) The phony war

I had to swap in my own existing modification of 'make_raw_alos.pl' that gives files named by their 8-digit dates by default, as indeed StaMPS likes. The regular version didn't seem to want to do that, and so the 'step_slcs_alos' step threw up a load of errors in passing. Once the switch was made, all went swimmingly – a master SLC image was generated, I picked out my area of interest (I used the mdx image viewer that comes with ROI_PAC to find the line and pixel numbers for the bounds), and was able to crop the master SLC image down to the bare essentials.

The next step, 'make_slcs_alos', also ran without complaints. I now had a processed SLC image for every date in my dataset. I was feeling so productive!

3) step_master_timing crashes with an error

FFS.

4) Manually fix the master date and time

After manually editing the date in the 'master.res' file to what I thought was the time it was after (I used the time listed as 'FIRST_CENTER_xxxx' for the hour, minute and second, and the fraction of the second that actually was given in 'master.res'), I actually got 'step_master_timing' to run. This was already a great improvement.

5) Hack the roipac2doris script to use the 'first center' time

I got times in the right format, at least!

6) No images matched

Bah.

7) Make a manual match and re-run Doris

Doesn't help.

8) Try again without step_master_timing

The images don't exactly match well, but they do match.

9) Now a manual timing correction

Crashed, so I ran step_master_timing.

Followed the instructions, but I couldn't see a match.

10) Looked at the DEM match in detail

The shift had hit a bound. Maybe the timing problem wasn't so well fixed after all.

11) Hack the roipac2doris script again (properly, this time)

Right. After some hacking around, swearing at the monitor, reading some man pages, swearing some more, and after a period of intense concentration, finally getting somewhere, I found that two lines containing the BSD 'date' command needed to be edited to account for the new format:

set UTCday = `date -juf "%Y %m %d %H %M %S" "$YEAR $MONTH $DAY 0 0 0" +"%s"` 

set UTC1 =  `date -juf "%s" "$UTCtime" +"%d-%m-%Y %T"`$AZIT1_DS

So, find 'em and edit away.

12) Delete all the SLCs and start again

When in doubt, purge.

13) Find that MATLAB has changed the default save convention

One last bonus crash! You need to make the following changes to account for the fact that new(ish) MATLAB doesn't save files like it used to...

ps_load_initial.m, line 193:

save(phsavename,'ph','-v7.3');

ps_est_gamma_quick.m, line 285:

save(pmname, 'ph_patch', 'K_ps', 'C_ps', 'coh_ps', 'N_opt', 'ph_res', 'step_number', 'ph_grid', 'n_trial_wraps', 'grid_ij', 'grid_size', 'low_pass', 'i_loop', 'ph_weight', 'Nr', 'Nr_max_nz_ix', 'coh_bins', 'coh_ps_save', 'gamma_change_save', '-v7.3') 

And hopefully now it should work. [Although, there are no guarantees that you will see any exciting signals in your data, of course...]