Background

Working in air-gapped environment means not having the luxury to rely on respective application updater to check for newer version automatically. So I needed to manually download the installers from time to time, which is quite time-consuming and a mundane task.

I usually automate repetitive and mundane work, and I have always wanted to automate this process, but I wasn't able to figure out an effective way to do so because (as far as I know) there is no standardized way of grabbing the download URL from various websites. Let's take a look at some samples.

[
  {
    "application": "7zip.7zip",
    "download_url": "https://www.7-zip.org/a/7z2200-x64.exe"
  },
  {
    "application": "Docker.DockerDesktop",
    "download_url": "https://desktop.docker.com/win/main/amd64/81317/Docker%20Desktop%20Installer.exe"
  },
  {
    "application": "Git.Git",
    "download_url": "https://github.com/git-for-windows/git/releases/download/v2.37.0.windows.1/Git-2.37.0-64-bit.exe"
  },
  {
    "application": "GitExtensionsTeam.GitExtensions",
    "download_url": "https://github.com/gitextensions/gitextensions/releases/download/v3.5.4/GitExtensions-3.5.4.12724-65f01f399.msi"
  },
  {
    "application": "Google.Chrome",
    "download_url": "https://dl.google.com/dl/chrome/install/googlechromestandaloneenterprise64.msi"
  }
]

There is no fixed templates / format to the URL, and there isn't a way to query for the download URL except for some like Cypress that provides a json file with the version and URL.

Ideas

This thing has always been at the back of my mind for a number of years, and I imagine maintaining the URL myself like such

{
    "application": "Google.Chrome",
    "version": "103.0.5060.66",
    "download_url": "https://dl.google.com/dl/chrome/install/googlechromestandaloneenterprise64.msi"
}

But I would need to manually update this as well, and doing so, would take even longer time than having to visit each website to download the installers myself. Hence, I didn't pursue the option since it didn't quite value-add for me.

Some other idea I had was to write scripts using cypress to navigate through the page to download various applications, but this will break easily if the website gets updated / changed. So the maintenance cost is high as well.

Winget

That is, until I came across winget. winget is a package manager for windows from Microsoft, it allows user to easily manage applications via winget just like you would on Linux using apt-get/yum.

Listing all the installed applications on my machine would look like

╰─ winget list --source winget
Name                                        Id                                           Version             Available
-----------------------------------------------------------------------------------------------------------------------
StarUML                                     MKLabs.StarUML                               4.1.6
P3X Redis UI                                PatrikLaszlo.P3XRedisUI                      2022.4.116          2022.4.126
Visual Studio Build Tools 2017              2fd33d5a                                     15.9.28307.1033

An example of a manifest for StarUML would look like

╰─ winget show "MKLabs.StarUML"
Found StarUML [MKLabs.StarUML]
Version: 4.1.6
Publisher: MKLabs Co.,Ltd.
Publisher Support Url: https://staruml.io/support
Author: MKLabs Co.,Ltd
Moniker: staruml
Description: A sophisticated software modeler for agile and concise modeling.
Homepage: https://staruml.io/
License: Proprietary
License Url: https://staruml.io/eula
Copyright: Copyright (c) 2021 MKLabs Co.,Ltd.
Installer:
  Type: nullsoft
  Download Url: https://staruml.io/download/releases-v4/StarUML%20Setup%204.1.6.exe
  SHA256: dd894019785afc4a49c1b18593ed2e0059ed28b6ce3bf85da0415da59a7a3d6d

While it is easy to list / install / upgrade the application via winget-cli, it is not as easy to parse the content as it is not a native PowerShell object, nor can it be exported to a json format. There are issue and discussion in GitHub tracking this closely, and I hope that these will be supported soon.

Nevertheless, I spent some time recently to write a simple PowerShell script to parse it, and it worked quite nicely.

Solution

I am not well verse in PowerShell script, so while it works, it may not be the best way to write it. I don't really care so much since it does its job as far as I'm concern.

Referring back to the StarUML manifest above. The key is to extract out the URL, and then download it.

This is the command to run to extract the URL

$url=winget show "MKLabs.StarUML" | Select-String "Download Url: " -NoEmphasis | Out-String | % { $_.Trim() } | % { $_.SubString(14) }

Let's take a look at how this command unfolds.

I will be showing the input and output in a single codebox

Let's fetch the manifest of the application

> winget show "MKLabs.StarUML"

Found StarUML [MKLabs.StarUML]
Version: 4.1.6
Publisher: MKLabs Co.,Ltd.
Publisher Support Url: https://staruml.io/support
Author: MKLabs Co.,Ltd
Moniker: staruml
Description: A sophisticated software modeler for agile and concise modeling.
Homepage: https://staruml.io/
License: Proprietary
License Url: https://staruml.io/eula
Copyright: Copyright (c) 2021 MKLabs Co.,Ltd.
Installer:
  Type: nullsoft
  Download Url: https://staruml.io/download/releases-v4/StarUML%20Setup%204.1.6.exe
  SHA256: dd894019785afc4a49c1b18593ed2e0059ed28b6ce3bf85da0415da59a7a3d6d

Use Select-String to search for the text we want

> winget show "MKLabs.StarUML" | Select-String "Download Url: " -NoEmphasis

  Download Url: https://staruml.io/download/releases-v4/StarUML%20Setup%204.1.6.exe

Convert output into String, otherwise, we can't apply String operation later on

> winget show "MKLabs.StarUML" | Select-String "Download Url: " -NoEmphasis | Out-String


  Download Url: https://staruml.io/download/releases-v4/StarUML%20Setup%204.1.6.exe

The output look exactly the same as before, but in String format. If we don't, when we apply String operation, it will throw the following error

InvalidOperation: Method invocation failed because [Microsoft.PowerShell.Commands.MatchInfo] does not contain a method named 'Trim'.

Apply Trim operation to remove all whitespaces

> winget show "MKLabs.StarUML" | Select-String "Download Url: " -NoEmphasis | Out-String | % { $_.Trim() } 

Download Url: https://staruml.io/download/releases-v4/StarUML%20Setup%204.1.6.exe

% is a shorthand for ForEach-Object

Apply SubString operation to grab only the actual URL

> winget show "MKLabs.StarUML" | Select-String "Download Url: " -NoEmphasis | Out-String | % { $_.Trim() } | % { $_.SubString(14) }

https://staruml.io/download/releases-v4/StarUML%20Setup%204.1.6.exe

And there we have it!

To download, we can use Invoke-WebRequest

Invoke-WebRequest -URI https://staruml.io/download/releases-v4/StarUML%20Setup%204.1.6.exe -OutFile ./staruml.exe

But it is very slow in download, and we have to specify explicit filename. So let's use curl instead

$default_download_dir="./_winget_applications"
$url=https://staruml.io/download/releases-v4/StarUML%20Setup%204.1.6.exe

# -L: follow redirect
# -O -J: we want to retain the remote filename instead of constructing our own
# see https://daniel.haxx.se/blog/2020/09/10/store-the-curl-output-over-there
# --create-dirs: if not exist
# --silent: do not show progress
curl -L $url -O -J --output-dir $default_download_dir --create-dirs --silent

So to recap, what we have done is to

grab the URL from the manifest
use curl to download the file

And what we have to do is to repeat the process for a list of applications. That's easy, just put it in a loop, and we're done.

For the complete script, please refer to my GitHub repository.

Conclusion

We looked at the step-by-step process of how I automate the mundane task of downloading application installers using PowerShell script with the help of winget-cli. However, this is not a foolproof solution as there are still a number of applications that have yet to onboard to winget, and so I will not be able to use this method to download those installers automatically.

I also hope to extend this automation to IDE plugins as well, which I have done so in the past for vscode but I now realize that there are much easier way to achieve the same goal.

Source Code

As usual, full source code is available in GitHub.

Joseph Gan's Blog