Battle plan for dropping `HttpServer`

Developer
1

Background

The HttpServer was added back when there was no stand-alone webserver for .NET. It was chosen because it was a small-ish project with a simple interface. Sadly it was never maintained, and I ended up “saving” it from oblivion when Codeplex was shut down.

The entire codebase in there is pretty typical of that era with very few things one would expect from a modern webserver (CORS, Domain, WebSockets, Dependency Injection, etc). @Jojo-1000 recently discovered that it even has its own (broken) implementation of UrlDecode.

I expect there to be many more problems hiding in various places. Fortunately some work was started by @tsuckow on creating a new webserver implementation using Kestrel and this was picked up by @npodbielski.

I have been looking at that combined work and I think it is important that this transition is done ASAP, so this post is an attempt to make a work plan for what should happen.

The current state is a mixed bag with some endpoints supported, some not, and most are wrapping existing functionality. My end-goal is to fully remove HttpServer and rely only on Kestrel, while providing a code-base that is familiar to ASP.NET developers.

I have made a map of all endpoints that needs to be implemented, so I will be deleting them and then copy back the code into the Kestrel functions, adapting as needed.

As always, input and help is much appreciated!

Proposed plan

My current overview of the task is listed below, and I will update as I progress:

  • Delete all references to HttpServer, including the endpoints
  • Get the Kestrel server starting
  • Introduce the previous commandline options and map them to Kestrel
  • Implement endpoints, update as needed
    • /api/v1/acknowledgements - Duplicati.Server.WebServer.RESTMethods.Acknowledgements
      • GET /api/v1/acknowledgements
    • /api/v1/backup - Duplicati.Server.WebServer.RESTMethods.Backup
      • GET /api/v1/backup
      • PUT /api/v1/backup
      • POST /api/v1/backup
      • DELETE /api/v1/backup
    • /api/v1/backupdefaults - Duplicati.Server.WebServer.RESTMethods.BackupDefaults
      • GET /api/v1/backupdefaults
    • /api/v1/backups - Duplicati.Server.WebServer.RESTMethods.Backups
      • GET /api/v1/backups
      • POST /api/v1/backups
    • /api/v1/bugreport - Duplicati.Server.WebServer.RESTMethods.BugReport
      • GET /api/v1/bugreport
    • /api/v1/captcha - Duplicati.Server.WebServer.RESTMethods.Captcha
      • GET /api/v1/captcha
      • POST /api/v1/captcha
    • /api/v1/changelog - Duplicati.Server.WebServer.RESTMethods.Changelog
      • GET /api/v1/changelog
    • /api/v1/commandline - Duplicati.Server.WebServer.RESTMethods.CommandLine
      • GET /api/v1/commandline
      • POST /api/v1/commandline
    • /api/v1/filesystem - Duplicati.Server.WebServer.RESTMethods.Filesystem
      • GET /api/v1/filesystem
      • POST /api/v1/filesystem
    • /api/v1/help - Duplicati.Server.WebServer.RESTMethods.Help
      • GET /api/v1/help
    • /api/v1/hyperv - Duplicati.Server.WebServer.RESTMethods.HyperV
      • GET /api/v1/hyperv
    • /api/v1/licenses - Duplicati.Server.WebServer.RESTMethods.Licenses
      • GET /api/v1/licenses
    • /api/v1/logdata - Duplicati.Server.WebServer.RESTMethods.LogData
      • GET /api/v1/logdata
    • /api/v1/mssql - Duplicati.Server.WebServer.RESTMethods.MSSQL
      • GET /api/v1/mssql
    • /api/v1/notification - Duplicati.Server.WebServer.RESTMethods.Notification
      • GET /api/v1/notification
      • DELETE /api/v1/notification
    • /api/v1/notifications - Duplicati.Server.WebServer.RESTMethods.Notifications
      • GET /api/v1/notifications
    • /api/v1/progressstate - Duplicati.Server.WebServer.RESTMethods.ProgressState
      • GET /api/v1/progressstate
    • /api/v1/remoteoperation - Duplicati.Server.WebServer.RESTMethods.RemoteOperation
      • GET /api/v1/remoteoperation
      • POST /api/v1/remoteoperation
    • /api/v1/serversetting - Duplicati.Server.WebServer.RESTMethods.ServerSetting
      • GET /api/v1/serversetting
      • PUT /api/v1/serversetting
    • /api/v1/serversettings - Duplicati.Server.WebServer.RESTMethods.ServerSettings
      • GET /api/v1/serversettings
      • PATCH /api/v1/serversettings
    • /api/v1/serverstate - Duplicati.Server.WebServer.RESTMethods.ServerState
      • GET /api/v1/serverstate
      • POST /api/v1/serverstate
    • /api/v1/systeminfo - Duplicati.Server.WebServer.RESTMethods.SystemInfo
      • GET /api/v1/systeminfo
    • /api/v1/systemwidesettings - Duplicati.Server.WebServer.RESTMethods.SystemWideSettings
      • GET /api/v1/systemwidesettings
    • /api/v1/tags - Duplicati.Server.WebServer.RESTMethods.Tags
      • GET /api/v1/tags
    • /api/v1/task - Duplicati.Server.WebServer.RESTMethods.Task
      • GET /api/v1/task
      • POST /api/v1/task
    • /api/v1/tasks - Duplicati.Server.WebServer.RESTMethods.Tasks
      • GET /api/v1/tasks
    • /api/v1/uisettings - Duplicati.Server.WebServer.RESTMethods.UISettings
      • GET /api/v1/uisettings
      • POST /api/v1/uisettings
      • PATCH /api/v1/uisettings
    • /api/v1/updates - Duplicati.Server.WebServer.RESTMethods.Updates
      • POST /api/v1/updates
    • /api/v1/webmodule - Duplicati.Server.WebServer.RESTMethods.WebModule
      • POST /api/v1/webmodule
    • /api/v1/webmodules - Duplicati.Server.WebServer.RESTMethods.WebModules
      • GET /api/v1/webmodules
  • Testing
  • Revisit login/auth/CSRF
2

To this point I just want to comment that right now even get requests require a csrf token. In the updated Angular implementation this isn’t supported by the built-in functions. Usually only non-safe methods (that modify state) have this protection. I would suggest to address that with this update.

GET requests can potentially leak CSRF tokens at several locations, such as the browser history, log files, network utilities that log the first line of a HTTP request, and Referer headers if the protected site links to an external site.

3

A current annoyance (to me) is that if I run the same web browser to two Duplicati on different ports, I get:

Failed to read backup defaults: Missing XSRF Token. Please reload the page

or maybe similar messages. I can stop this by putting one Duplicati on localhost and one on 127.0.0.1 because that causes Edge (so presumably Chrome) to consider them different, having their own cookies.

This setup is a bit unusual currently, but I don’t know if any of the Duplicati Inc. work will get into it further…

4

That could be fixed for example by adding the port to the cookie name. Then they would still be shared, but each instance has its separate cookie. Maybe only if the default port is not used.

1 Like
5

This looks like it only applies for CSRF applied as query strings, not headers, which is recommended just below the quote:

Since requests with custom headers are automatically subject to the same-origin policy, it is more secure to insert the CSRF token in a custom HTTP request header via JavaScript than adding a CSRF token in the hidden field form parameter.

I agree and I think the whole setup has to be updated to better reflect best practices 2024.

6

Ok, so I made “some” progress: Remove httpserver by kenkendk · Pull Request #5227 · duplicati/duplicati · GitHub

7

I have now made an overhaul of the security of the webserver:

Some highlights:

  • Empty/no password is no longer permitted
  • Random password assigned on first start
    • Tray-icon can generate a signin token that grants access without knowing the password
    • Webserver will output a link with a signin token on startup
    • Both options can also use --webservice-password=... to (re-)set a password
  • Password is stored with PBKDF2 to prevent #5197
  • Using three levels of tokens, all based on JWT
    • SignIn: very short lifetime; only supports granting a login
    • Access: The bearer token that grants access to the API; short-lived, never persisted in browser
    • Refresh: A token that can be exchanged for an access token; long-lived, only works on a single endpoint; using token counters to prevent re-use
  • Removed XSRF as the system no longer relies on automatic credentials (no auth/session cookies)

The system now stores the encryption key for the JWT in the database, which means it is still vulnerable to the method described in #5197, but it does not expose the password. With the key it is possible to issue SignIn and Access tokens, and with some luck fake a refresh token.

I would like to fix this with a general approach to storing sensitive information in the database, so we can cover the remote destination credentials and other things as well.

I have added the method suggested by @Jojo-1000 with different cookie names for different host ports.

8

I have now gone through most of the functionality now, and it generally works.

There are still some cases where the parsing of arguments is slightly different and this causes “Direct restore” to fail.

@npodbielski I have temporarily disabled websockets because it was not correctly relaying all status messages, but the implementation looks really nice and simple.

9

Hey sorry for being silent about this. I had very busy period at work (whole May and half of June) and now I am on vacation till the end of June. Maybe I will be able to do something next week - I am not sure. It is finally nice whether so I have a lot of stuff to do in the garden also - it is better to spend some time in the sun instead of indoor watching computer screen :smiley:

I did not moved all of the longpolls to the websockets. Should work in parrallel - maybe I did made some mistake sorry about this.

10

Sounds amazing :sun_with_face:

I can just report what I see, without debugging further:

  • Browser “connects” and seems connected, but does not seem to send the connected event
  • The backend never triggers the connect and appears to have no connected sockets
  • Stopping the server triggers a disconnect event in the browser

I might have broken things while I was restructuring the code and adding authentication :confused:

11

Hi, finally got some free time to sit on it.

I see you working on feature/remove-httpserver branch now? Is that right?

12

Yes! It is ready for a merge, but I might remove some more.

13

So I poke around a bit and it seems that JWT config is incorrect.

System.InvalidOperationException: IDX20803: Unable to obtain configuration from: ‘https://duplicati/.well-known/openid-configuration’. Will retry at ‘3.07.2024 16:54:49 +00:00’. Exception: 'System.IO.IOException: IDX20804: Unable to retrieve document from: ‘[PII of type ‘System.String’ is hidden. For more details, see Bing]’.
—> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
—> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure: RemoteCertificateNameMismatch
at System.Net.Security.SslStream.SendAuthResetSignal(ReadOnlySpan`1 alert, ExceptionDispatchInfo exception)
at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptions sslAuthenticationOptions)
at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte reAuthenticationData, CancellationToken cancellationToken)
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)

Can you explain a bit how does it work? Or how it supposed to work?
I do not see anything IdP under http://duplicati.com
There is something under https://duplicati-oauth-handler.appspot.com/ I think… but it does not seem to be used for JWT validation.

14

I am bit worried about that JWT implementation. I am not sure what is IdP in that case but if you want Duplicati still to be Open Source and possible to self host I would expect it to be configurable and it should be possible to connect to my IdP i.e. Keycloak, Gitlab or Google or whatever.

In that case it would be nice to open configuration page and be able to configure JWT IdP to my own.

Or via docker configuration - I am not sure what is your end goal approach here.

15

The general idea is that the startup process generates a random key and this is used to issue and verify JWT. The configuration is thus unique for each machine and stored in Duplicati-server.sqlite in the table Option under the id jwt-config (this setting should be encrypted, but for now it is not).

It is not integrated with any IdP and does not need one. How did you get the error message above?

When the tray-icon opens the browser, it directs to /signin.html?token=... where the ... value is a freshly generated token with a short lifetime. The browser submits this and obtains a refresh token that is stored as a httponly cookie (potentially secure as well).

For server-based setups, the console log will contain the link on startup, but you can also use --webservice-password to set a known password and log in with the password.

The refresh token is exchanged for an access token via /api/v1/auth/refresh and the access token is applied to all calls via the standard Authorization: Bearer ... header.

If the access token expires, the AppService.js code will detect the 401 response, and request a new access token. If this fails, you are directed to the /login.html page.

In short there are three types of tokens (as described above) that has different functions.

Each call uses the standard JWT library and verifies that the token is valid and issued with the random key assigned to the server. If you start with --webservice-reset-jwt-config it will rewrite the JWT config in the database, meaning it assigns a new random key, which invalidates all existing tokens.

There is nothing there, it is not configured for use with any external domain or service.

I certainly want Duplicati to be open source, and self-host-able. It might make sense to support external OAuth-based IdPs, but in this implementation it is using only the JWT tokens, not the OAuth infrastructure.

The goal is simply to prevent unwanted websites or applications with access to localhost to manipulate the server. Prior to the JWT setup, anyone running without a password would have a weak protection in the browser, due to XSRF, but would be completely open to a non-browser application running on the machine.

Instead of inventing a new setup, I have used the JWT standard to issue and validate tokens. This approach is simpler, and more robust, while working great with the browser.

16

Oh I did not do much. I just enabled WebSocket and was debugging why it does not work. I am not sure yet if this is problem with Authentication, Authorization and JWT configuration or with WebSocket.

Anyway as far as I can tell, for now, this is during Authentication, when JWT handler is trying to validate the token send via query parameter during WebSocket connection handshake.
Here is whole stack:

System.InvalidOperationException: IDX20803: Unable to obtain configuration from: ‘https://duplicati/.well-known/openid-configuration’. Will retry at ‘3.07.2024 19:37:44 +00:00’. Exception: 'System.IO.IOException: IDX20804: Unable to retrieve document from: ‘[PII of type ‘System.String’ is hidden. For more details, see Bing]’.
—> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
—> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure: RemoteCertificateNameMismatch
at System.Net.Security.SslStream.SendAuthResetSignal(ReadOnlySpan1 alert, ExceptionDispatchInfo exception) at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptions sslAuthenticationOptions) at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken) at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem) at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at Microsoft.IdentityModel.Protocols.HttpDocumentRetriever.SendAsyncAndRetryOnNetworkError(HttpClient httpClient, Uri uri)
at Microsoft.IdentityModel.Protocols.HttpDocumentRetriever.GetDocumentAsync(String address, CancellationToken cancel)
— End of inner exception stack trace —
at Microsoft.IdentityModel.Protocols.HttpDocumentRetriever.GetDocumentAsync(String address, CancellationToken cancel)
at Microsoft.IdentityModel.Protocols.OpenIdConnect.OpenIdConnectConfigurationRetriever.GetAsync(String address, IDocumentRetriever retriever, CancellationToken cancel)
at Microsoft.IdentityModel.Protocols.ConfigurationManager1.GetConfigurationAsync(CancellationToken cancel)'. ---> System.IO.IOException: IDX20804: Unable to retrieve document from: '[PII of type 'System.String' is hidden. For more details, see https://aka.ms/IdentityModel/PII.]'. ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure: RemoteCertificateNameMismatch at System.Net.Security.SslStream.SendAuthResetSignal(ReadOnlySpan1 alert, ExceptionDispatchInfo exception)
at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptions sslAuthenticationOptions)
at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte reAuthenticationData, CancellationToken cancellationToken)
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
— End of inner exception stack trace —
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)
at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) at Microsoft.IdentityModel.Protocols.HttpDocumentRetriever.SendAsyncAndRetryOnNetworkError(HttpClient httpClient, Uri uri) at Microsoft.IdentityModel.Protocols.HttpDocumentRetriever.GetDocumentAsync(String address, CancellationToken cancel) --- End of inner exception stack trace --- at Microsoft.IdentityModel.Protocols.HttpDocumentRetriever.GetDocumentAsync(String address, CancellationToken cancel) at Microsoft.IdentityModel.Protocols.OpenIdConnect.OpenIdConnectConfigurationRetriever.GetAsync(String address, IDocumentRetriever retriever, CancellationToken cancel) at Microsoft.IdentityModel.Protocols.ConfigurationManager1.GetConfigurationAsync(CancellationToken cancel)
— End of inner exception stack trace —
at Microsoft.IdentityModel.Protocols.ConfigurationManager1.GetConfigurationAsync(CancellationToken cancel) at Microsoft.IdentityModel.Protocols.ConfigurationManager1.GetBaseConfigurationAsync(CancellationToken cancel)
at Microsoft.IdentityModel.JsonWebTokens.JsonWebTokenHandler.ValidateTokenAsync(JsonWebToken jsonWebToken, TokenValidationParameters validationParameters)

as you can see it originates in JsonWebTokenHandler.ValidateTokenAsync. It just tries to validate if token was issued using correct signing keys. Public keys are usually available either with token itself or in IdP web server. Asp.Net probably is just trying to find them and to do that tries to read .well-known/configuration under token iss claim (issuer domain).

17

Oh, and thanks for detailed explanation. I will be able to fix the issue with that :slight_smile:

18

ah no I was wrong. Above error is caused by periodic check by JWT config.

Anyway… origin configuration was incorrect.

I pushed few changes to branch on my fork: npodbielski/feature/remove-httpserver

19

Hmm, I have not seen it in any of my runs.

Awesome! I have a fix for the origin check that keeps hostnames and origins separate:

Only thing that appears to be a bit flaky is the reconnect. I cannot see that the 401 is reported to the browser if the user is not logged in, so it keeps retrying.

@npodbielski Is the intention to use the regular HTTP call to /api/v1/serverstate as the handler for checking for auth-expiration? If so, I can re-work the JS to use this logic.

I can only see that the websocket call is being retried, but this fails for some reason, even with a valid token. Not sure why, but the user is rejected as un-authenticated, even with a valid token. Reloading the page fixes everything.

Previous logic was to check if the event number has changed after a reconnect and then reload the page in case it was changed (no change == same state, change == missed update event). I think this can be re-added to the WS code.

20

No, you are right this does not makes sense. I will try to fix it today.

No, this does not makes sense also.

Does refresh will help?
During debug there is a message

Initial signin url: http://localhost:8200/signin.html?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ0eXAiOiJTaWduaW5Ub2tlbiIsInNpZCI6InNlcnZlci1jbGkiLCJuYmYiOjE3MjAxNjI5NTgsImV4cCI6MTcyMDE2MzI1OCwiaXNzIjoiaHR0cHM6Ly9kdXBsaWNhdGkiLCJhdWQiOiJodHRwczovL2R1cGxpY2F0aSJ9.i2Suhv_ACWJQtJConAewytSsFYHAC0jqe_8GAXhYr_Q

so I thought that intention is to open it via Gui icon. In this case refreshing will not help. Right?