A wild techblog article appeared! Today we're going to take a looksie at our new download and install experience which we rolled out a few months ago.

Why did you make a new downloader/installer?

Three reasons: downloading slowly sucks, people don't read, and it's our first big funnel dropoff that we control.

Everyone hates waiting for a download to finish—we hear stories from some of our users that they have to wait up to an entire day to download the game, and it isn't because their internet is crap (we can't do anything about crap internet though). Unfortunately, our current download system is extremely simplistic and does not let us deliver the client quickly to players all over the world at a reasonable cost.

We frequently got technical support requests from users that don't read the download and installation instructions on the download page. They forget to install the prerequisites or Windows Defender yoinks the game files away.

Downloading and installing the game is the first "real" interaction with the game the user experiences. We looked at some of our internal statistics and noticed that on average 20% of accounts that register an account in a given month never actually log into the game and create a character. Long download times can be a huge turnoff for prospective players to check out our server.

We are considering automatically purging accounts that register but don't play after some delay, which should also free up some common usernames that people try to get but can't.

What did we have before?

Previously, the download archive is replicated across several web servers that also run the website and other online features, most of which are located in a datacenter outside of San Fransisco (which ironically is across the country from the game server, in New York). We also had two mirror servers in the UK and Singapore that only served static files and the download archive. When you clicked the download button, the request was routed through Cloudflare and then load balanced across our web servers in SF/UK/SP. Because the archive is so big, CloudFlare will not cache it on their edge servers - the servers that are close to you that you hit first. They currently are caching our website assets, which is why our site loads so quickly regardless of where you are in the world.

Each mirror webserver costs a paltry minimum of $5 a month + bandwidth costs (our primary servers are on $15/month plans), but in order to serve more traffic and get closer to the geographical regions we serve, we have to spin up servers and so that cost balloons quickly (not to mention sysadmin time I have to spend when they decide to misbehave). We'd rather spend your donations on more interesting or important things (and upgrading our game server).

Cheap Chocolate Chunks

I've used Backblaze B2 storage in the past for a university project, and initially considered hosting the archive with them and utilizing their content delivery network (CDN). Testing, however, showed pretty abysmal download speeds even on my own internet, plus the bandwidth costs would become prohibitive (and easily abused by an attacker) so that idea was scrapped. However, they do have a partnership with Cloudflare where transfers from B2 to Cloudflare are free. Using this partnership gave slightly faster speeds, but again because our archive is so big, we wouldn't be able to take advantage of Cloudflare's edge caching.

The answer to that, of course, is to chunk your download into smaller pieces. We actually even do that as an alternative download, where we split the download into roughly 1 GB chunks. However, Cloudflare's max cachable size is 512 MB, and the larger the file the less likely the file will stay in cache, and we want to have as high of a cache hit rate as we can possibly get (for speed and cost reasons).

There's several ways to chunk a big file down into smaller ones, but the more chunks you make the more of a burden it is for the user (and more likely that they're going to mess it up or not even try at all, which would be counterproductive to solving our 3 issues). We can avoid this by making a downloader program that is responsible for downloading the chunks and putting them together.

Another benefit of chunking into small pieces is that many ISPs do something called "burst speeds," where for a brief period of time at the start the ISP gives a connection faster download speed than what you pay for. They do this to make their internet feel faster, since most files you download from the internet are small (think icons on a news website) and the speed ends up averaging out anyway. We can take advantage of this by using small enough chunks that download quickly enough during their burst window to mostly complete. For ISPs that do this aggressively, you can actually end up downloading faster than your sticker speed.

Rito Pls

So, Riot Games (of League of Legends fame) not too long ago put out a techblog post that described how their patcher works, and it's definitely an interesting read. And unlike the rest of the out-of-game client, their patcher is actually insanely good.

So, tl;dr, we're basically using similar tech to power our new downloading experience and eventually power patching as well. Reading over their blog post will give a pretty good overview of what we're doing.

The Breakdown

Our new downloader downloads a manifest file that describes all the files in a Project Duck installation and all the chunks that constitute each file. Additionally, the manifest contains a list of bundles and the chunks that each bundle contains. Each bundle is roughly 32 MB so they can be cached by Cloudflare.

The downloader then downloads up to 8 bundles in parallel and writes the chunks in each bundle to their corresponding position in the correct files. This also means we can do some deduplication, since we only need to download a chunk (in a bundle) once even though its used by more than one file.

Impact

When we released the new installer, many players were blown away at how fast they could download a clean build. We saw improvements across the board - players with fast internet connections could download in mere minutes and players with slow internet connections downloaded up to 3x faster (which for those that previously had download times in the overnight range can now download in a few hours). We can comfortably say we have the fastest download experience out of any DN server.

The registration-to-character creation funnel has also been improved and we are seeing more players actually make it to the game, which is a key win here.

We also saw a significant drop in technical support questions related to not having the prerequisites (.NET and VC++) or the game getting yeeted by Windows Defender (the life of a pserver client...).

However, we did run into a few new issues:

• Users would run the installer from within the download zip, which confused the installer and caused it to default to /Windows/System32 (oops)
• Users would run it from their downloads folder and spray game files all over their downloads folder
• Sometimes users that run the installer again (to reinstall or whatever) are unable to launch the installer. This is a manifestation of this known .NET Core bug which is expected to be mitigated in .NET Core 5. To fix this issue, the user simply needs to delete the %temp%/.net/ProjectDuckInstaller folder and relaunch
• Users that entirely lose internet connection (briefly or prolonged) are unable to continue downloading and have to restart from scratch

We've since updated the installer to refuse to install to system32 or a downloads folder and instead default to %appdata%/ProjectDuckDN when the installer is unable to assert that the target directory is safe.

Next Steps

We're super happy with the outcome of the new installer and there are a few things left for us to do with the current version:

• Resumable downloads (both to handle network hiccups and to allow users to close the installer and resume installing later)
• Download speed throttling (because sometimes you don't want to saturate your internet connection)
• Creating a desktop shortcut to the launcher after installing (some users have "lost" where they installed the game)

However, we do want to go bigger. Here's some design sketches (so totally WIP and not final!) for the new installer, patcher, and launcher:

We're really excited to bring you new features and tech to improve your experience both in and out of game!