Pre Caching macOS Software

Icon for Caching Server
Caching Server

I often find myself in a spot where I’m always re-downloading and re-caching iOS IPSW files because they’re infrequently being downloaded from Apple. This is pretty frustrating when it comes time to downloading macOS installers, iWork/iLife apps, or IPSW’s for the infrequent restoring of an iOS device in Apple Configurator. This is a feature sorely lacking in Apple’s macOS Caching Server; so until that feature is implemented, we make do…

Charles from krypted.com  has a great post about how to pull cacheable assets through the Caching Server, so using this method, we can figure out where those Mac App Store apps are coming from, and pre cache those!

Lets take a look at the Caching Server logs found at /Library/Server/Caching/Logs/ – specifically Service.log

2016-10-10 10:14:45.829 #/lsyeQGL0tvE Received GET request by "MacAppStore/2.2" for /apple-assets-us-std-000001/Purple62/v4/ed/3d/8e/ed3d8e87-09da-2272-fc3a-b1678d8067a0/iyp5743666419479406275.pkg
2016-10-10 10:15:45.627 #E3oGY2esLpZE Received GET request by "itunesstored/1.0" for /apple-assets-us-std-000001/Purple20/v4/57/44/6b/57446bcb-a3b7-46ae-3307-788a3b5ef280/pre-thinned2492386531421197760.thinned.signed.dpkg.ipa
2016-10-10 10:15:50.174 #E3oGY2esLpZE Served all 2.9 MB of 2.9 MB; 0 bytes from cache, 2.9 MB stored from Internet, 0 bytes from peers
2016-10-10 10:17:35.870 #ziqCwn3ZEdaC Received GET request by "itunesstored/1.0" for /iOS7.1/031-0753.20140310.D1cKf/com_apple_MobileAsset_DictionaryServices_dictionary2/274d8f957d6bd64df440645d851c0dbd4deea358.zip
2016-10-10 10:17:42.328 #ziqCwn3ZEdaC Served all 28.0 MB of 28.0 MB; 28.0 MB from cache, 0 bytes stored from Internet, 0 bytes from peers

In the example above, we see that the Caching server has received a GET request from a Mac App Store client, and there is a package file being requested; the URL however is incomplete.

Looking at active requests on our proxy server at the time the request went through, the complete requested URL is found:

by kid3 {
...
uri http://osxapps.itunes.apple.com/apple-assets-us-std-000001/Purple62/v4/ed/3d/8e/ed3d8e87-09da-2272-fc3a-b1678d8067a0/iyp5743666419479406275.pkg
...
}

Bingo!

Now we can pull Mac App Store apps through the Caching Server, all we need to do is watch the Caching Server Service.log file for any requests for items that aren’t cached. The package URL’s are not human readable, so basically this involves watching the logs with something like tail -f /Library/Server/Caching/Logs/Service.log or watching the Caching Server service log in the Server app.

Here’s some URL’s I’ve grabbed so far (these have all been downloaded on macOS Sierra 10.12.0* and are “correct” as at 2016-10-10) – you can use curl or wget to grab these through your Caching Server:
*The iWork apps are macOS Sierra specific.

Pages 6.0
http://cache_server_url:PORTNUM/apple-assets-us-std-000001/Purple62/v4/8a/ee/6e/8aee6e8b-e8cb-2434-b050-31dbbcc01974/daf974703926683564923.pkg?source=osxapps.itunes.apple.com

Keynote 7.0
http://cache_server_url:PORTNUM/apple-assets-us-std-000001/Purple71/v4/a6/96/42/a696423f-181c-fc2b-b572-3d3697146d47/hlz1727390940373748952.pkg?source=osxapps.itunes.apple.com

Numbers 4.0
http://cache_server_url:PORTNUM/apple-assets-us-std-000001/Purple71/v4/69/02/32/69023287-bd7a-ef14-0424-234d8fc589e4/mto6541029270492763328.pkg?source=osxapps.itunes.apple.com

GarageBand 10.1.2
http://cache_server_url:PORTNUM/apple-assets-us-std-000001/Purple30/v4/19/78/8b/19788bde-3172-3b98-8300-b8c4a9458bae/iat2506504784673372233.pkg?source=osxapps.itunes.apple.com

iMovie 10.1.2
http://cache_server_url:PORTNUM/apple-assets-us-std-000001/Purple20/v4/80/6d/9b/806d9b4e-776c-baae-574c-ed8afbc70acb/gyj6237528809531298180.pkg?source=osxapps.itunes.apple.com

Xcode 8.0
http://cache_server_url:PORTNUM/apple-assets-us-std-000001/Purple62/v4/ed/3d/8e/ed3d8e87-09da-2272-fc3a-b1678d8067a0/iyp5743666419479406275.pkg?source=osxapps.itunes.apple.com

macOS Server 5.2
http://cache_server_url:PORTNUM/apple-assets-us-std-000001/Purple62/v4/44/71/01/44710118-b2c9-1e31-73f6-fa7a0a26e594/wjs7031774084062486733.pkg?source=osxapps.itunes.apple.com

Substitute the cache_server_url for your Caching Server address, and the PORTNUM for the Caching Server port number, which can be found by running sudo serveradmin fullstatus caching. This process can be done for any of the Mac App Store apps, however there are a couple of small gotcha’s:

  • iOS 9 introduced App Thinning, this makes it difficult to pre cache iOS apps, as there are now many different downloadable assets for the one app.
  • These URL’s may change any time there is an app update.
  • Downloaded packages are not installable from the GUI or the CLI as they are encrypted.

You can grab Charles’ precache.py from here. This utility will pre cache iOS, watchOS, and tvOS Over the Air (OTA) updates, as well as the IPSW files for the same OS’s, it will also pre cache macOS installers for Mountain Lion through to current macOS Sierra release.

There is also the CacheWarmer utility available.

 

** Note **

So, for all those thinking to themselves, “Hey Carl, that’s cool, but why don’t I just download them from the Mac App Store when they come out?”
Well, that’s an entirely reasonable approach to take, until you’re in a situation where you need to make sure these apps stay in your Caching Server. By creating a script to download these apps, you can guarantee that they’ll be there when you need them. It also makes it pretty handy to get a Caching Server ready again after having to reset the cache after an asset becomes corrupted (which does happen), or in instances where Caching Server decides to clear the cache when it can’t contact Apple for registration.