I promised myself I would never write a file format without a version field, but here we are.
As part of upgrading Duplicati to .NET8, I discovered that the updater manifest file for Duplicati pointed to the “one golden zip file” that was used with .NET4. With .NET8 there is no golden zip file since we have different versions for each operating system, so I had to rework the contents of the file in an incompatible way. This presented a couple of problems:
- After trying to just enter the new content, I realized the key-length was too short and the file format didn’t support changing the key size.
- Worse: This was the first time I had to update the file format, and I realized there was no version field.
Well, well, well, if it isn’t the consequences of my own actions
With no version field, any update to the file format has to use some quirk to detect versions. If you’ve ever dealt with versioning for Zip files or Tar files you will have horror stories.
The format was quite crude, consisting of just the signature bytes (fixed-length) and then the JSON content. At a stretch, I could have attempted version detection by looking at the first position after the signature and seeing if the rest of the file was valid JSON.
Had I included a version field though, it might have been possible to update the file format, but with no built-in versioning and no support for the update in the current clients, the best approach seemed to be a new file format.
An interim solution
To preserve some compatibility, I have updated the clients to support a deprecation field. If this field is set, the clients will not attempt to download the package but show the update URL. Clients that do not support the field will request to update to the latest version with the old manifest, and then discover the deprecation field.
This leaves the new clients open for a new file format, but this time it would be really nice if future changes could be supported more smoothly. So, I went in search of another file format that would support my requirements, but came up with nothing.
New format requirements
The updater manifest has two parts: a signature and a payload, and this post focuses on the signature. I had some very basic requirements for the new file format:
- Versioning support
- Data payload is JSON
- Must be signed and verifiable with standard C# code
- Must use asymmetric private-public verification
- Must support multiple signing keys for rotating keys
The payload format being JSON is not that important, but since the rest of Duplicati relies heavily on JSON, it makes sense to stay consistent. The signing is important, as the updater tool may download and execute the files, so it must be verified that the trust placed in the original download extends to the update.
The original updater was designed to support hosting via unencrypted HTTP, back when LetsEncrypt was not a thing. Even though encrypted connections are now the standard, having the file signed protects against access to the storage server or update domains, so I still want this security.
Existing proposals
We could have signed the file format with GPG, but that would require GPG to be installed on all systems, and it is not bundled with Windows for example. I also wanted the signature to be embedded in the file, enabling a single file for more efficient network operations and avoiding issues with partial updates.
With these requirements I went looking for a “signed JSON” standard or format. I found one suggestion and another similar solution.
Both proposals aim to keep the JSON file readable by JSON libraries, even when signed. While I can understand the neatness in that, I do not like the modification part. One very basic rule in signing protocols is that you must sign everything or risk a weakness. That means that the signature must be calculated from the full source data. The suggestions I found essentially do this, but then modify the original data to inject the signatures. This is simple enough for the signing process that can look for the first-or-last }. But the verification is more involved, because it has to reproduce the original, unmodified JSON. Doing this in bytes is very error prone, as there are special cases to (potentially) handle.
One case is trailing comma handling, where the signature could be inserted before another element (requiring an extra comma to be inserted after) or it could be inserted last, in which case there is no extra comma to remove. Another special case is a JSON string that is an array as the outermost element, which will require some extra logic to add/remove an outer object.
New format suggestion: A spin on JWT
I wanted the payload and signature to be clearly separated to leave less room for error. One example of this would be JWT that relies on Base64 encoding not using the . character, and then splits the payload and signature with this “special” character. I liked this idea of having a marker that clearly separates the two components.
The JWT format is in itself not enough, as it supports only a single signing key. The key can be asymmetric, but using multiple keys would require having the same contents in each signed key. This is error prone as there is no guarantee that all payloads are identical, and JWT also can’t be read as a JSON file, but requires special parsing. However, the general idea of using Base64 to preserve whitespace, having a header and the . character as a separator can be used.
I settled on adding a versioned header to the JSON as a comment
Not all JSON libraries support comments, but by relying on a reserved header character (linefeed, 0x0A), the client can easily strip the comment before parsing. If the source is Windows, there may be a \r\n ending of the signature, but this whitespace belongs to the header, not the payload.
The file format would then look like this on the top level:
//SIGJSONv1: ...base64-header.base64-signature...\n
...payload...
The start marker //SIGJSONv1: is a “magic marker” that indicates this is a signed document. It is designed to look like a comment to have some support from JSON readers that don’t understand the format. The “magic marker” lines can be repeated to add additional signatures or signing methods. The markers are considered valid for versions 1 through 9, and clients MUST ignore versions that they do not support. Each line is an independent signature and may only contain the marker and the Base64-encoded components. A comment line that does not include the “magic marker” must be part of the payload, and no further signature detection is done. In other words: interleaving comments and signatures is not supported.
The payload can then be included directly as a stream, preserving encoding etc. The linefeed character ends the comment, and is also part of the header, not the payload. The header and signature are equivalent to the JWT format and follow the same rules, allowing extra fields, but at a minimum requires the alg and key parts:
[
{
"alg": "RS256",
"key": ...public key...
}
]
Note that there is no PKI or trust implied by having the public key in the signature, it is only to simplify the validation. By examining the headers, the client can identify which keys it trusts and validate with those.
Creating the signature
The header and the payload are combined as with JWT to calculate a signature:
HMAC_FUNC(
header_length
header +
payload
)
Unlike JWT, the contents are not Base64 encoded before the signature is calculated, and the . character is not part of the signature. Instead, the length of the header in bytes is encoded as a 32-bit unsigned value in network byte-order. The motivation for including the header length is to guard against cases where an attacker could move the boundary between the two, causing two different messages to validate with the same signature.
After the signature has been calculated, the header and signature are Base64 encoded (regular Base64, not Base64url) and joined with the . character to form the signature token:
'//JSONSIGv1: '
+ base64(header)
+ '.'
+ base64(signature)
+ '\n'
The construction is repeated for each signing key/method. As the signatures end with a linefeed character they can be safely concatenated, and the payload is concatenated at the end. The signing application should verify that the payload does not start with the signature, but otherwise treat the contents as a binary stream.
Although the file now looks like a text file, the signature verification is white-space sensitive, so the file should not be edited after being signed.
The deviation from JWT in terms of encoding is done here because the contents are not intended to be passed as URLs, but to be contained fully within the file. This makes it possible to skip the encoding steps and also to use the Base64 encoding with padding for easier decoding.
Resulting signature example
After building a signature, the resulting file will look something like this (shortened for readability):
//SIGJSONv1: W3siS2V5IjoiYWxnIiwiVmFsdW...x1ZSI6IlNJR0pTT052MSJ9XQ==.VsdpdDkDW...VtVDu1qkqhGXkcZj9w==
//SIGJSONv1: W3siS2V5IjoiYWxnIiwiVmFsdW...ZhbHVlIjoiU0lHSlNPTnYxIn1d.p0+KdWp5x...qXtkvgLVeGpd3sZS+Q==
{
"id": 5,
"extra": {
name: "test"
}
}
The updated file format was merged in April 2024. As always, your feedback and discussion are welcome.
Credits
Rebecca Dodd is a contributing writer on this article.