Kitten

Technical Manual sections

This is the order in which the Kitten Server handles a request. Middleware and Routes describe the steps, Initialisation (asynchronous) explains the order of the routes, and Errors and restarts covers the error handler.

Flowchart: The request pipeline

The Kitten Server adds middleware that acts on incoming requests before any routes do. The initialise() method calls addAllMiddlewareToApp() from src/middleware/index.ts right after it creates the Polka app.

Middleware does only what applies to every request, whatever its route. Decisions about a route (whether it responds to a WebSocket upgrade, whether it requires sign-in, and how it parses the request body) belong to the route that Polka runs: the route handler makes the first two, and the route itself parses the body. A unit test in tests/middleware/index.ts asserts the order of the middleware below, by function name, in development and in production.

Middleware often modifies the request and response objects that routes receive to inject functionality that is useful during authoring.

The list, below, is in the order the middleware runs in, which is the order Kitten adds it in. Polka runs middleware and routes in the order they are added, so middleware added earlier acts on the request earlier and can end the response before the rest runs.

Source code:

The Routes class creates the static and dynamic routes for all files at a given base path.

The default base path is that of the app or site that Kitten is serving, which Kitten stores in the basePath environment variable (process.env.basePath). The Kitten Server calls Routes.getRoutes() twice: once for Kitten’s own web app (at paths.KITTEN_WEB_APP_DIRECTORY_PATH) and once for your app. (See Initialisation (asynchronous) for the order in which it adds them.)

Routes.getRoutes(basePath = process.env.basePath)

The bulk of the work in instantiating server routes is split between the Routes class and the Files class it uses.

Files class

The Routes initialiser gets the project’s files from its Files instance, sorted by extension category type, and does two things:

  1. In development, listens for the file event dispatched by its Files instance and restarts on any change (see File changes).

  2. Creates one route for each file (and a page socket route for each page), refuses routes it cannot serve, and orders the rest. The getRoutes() method returns the ordered list, and the Kitten Server adds the routes to Polka in that order, whatever order Files found the files in.

The refuseUnservableRoutes() method throws one RouteTableError whose message has a line for each problem that routeTableProblems() in src/lib/routeTable.ts finds. It refuses two routes with the same method and normalised pattern, from any extensions (e.g., about.page.js and about.page.ts, zzšŸ”’.page.js and zz/index.page.js, or default.socket.js and a page’s own socket), with a line that mentions both files and the URL. It refuses each .connect.js or .connect.ts file, as Node never passes CONNECT requests to routes. The Kitten Server logs the message and exits with code 65 (systemExitCodes.dataError).

A normalised pattern (normalisedPattern() in src/lib/routeTable.ts) is the pattern in lower case, as regexparam matches without case, with its parameter names removed. So /:slug/ and /:name/ are the same normalised pattern, and so are /rooms/:any.socket and /rooms/:room.socket.

The comparePatterns() function orders two patterns by their normalised segments. At the first segment where they differ in kind, a literal segment comes before a parameter, a parameter before an optional parameter, and an optional parameter before a wildcard. A pattern that runs out of segments first comes first, unless its last segment is a wildcard or an optional parameter, in which case the longer pattern comes first (so /docs/*/edit/ comes before /docs/*). Static and dynamic routes are ordered together. Ties are broken by the byte order of the normalised pattern (so parameter names never decide precedence), then by putting HEAD routes first (as Polka runs GET routes for HEAD requests), then by the byte order of the method, the pattern, and the file path.

Route classes are determined according to file extensions (routeFilesFor() in src/lib/routeFiles.ts, with the extension lists in src/Utils.ts):

The routePatternFromFilePath() function in src/Utils.ts derives each route’s pattern from its file path, removing every šŸ”’. The exception is client scripts (.client.js and .client.ts), which the clientScriptPatternFromFilePath() function serves at their own path, without route parameters. In a socket route’s file name, [any] becomes a one-segment parameter that preserves the .socket suffix (e.g., rooms/[any].socket.js is at /rooms/:any.socket). A route’s method comes from its extension: get for pages, static files, socket routes, and page socket routes, and the HTTP method for HTTP routes.

The requiresSignIn() function in src/Utils.ts (which calls requiresSignInUnder() with process.env.basePath) reports whether a route requires sign-in: whether its file path, relative to the app (or to Kitten’s internal web app), contains šŸ”’. This ensures that an app in a folder whose path contains šŸ”’ doesn’t require sign-in everywhere.

All static routes are of type StaticRoute and expose a GET handler that depends on the file:

Kitten serves a .gz file with the Content-Encoding: gzip header and the content type of its name without the .gz extension (e.g., script.js.gz is served as JavaScript). If the request’s Accept-Encoding header doesn’t include gzip, connect-static-file passes the request on to the next handler.

Dynamic routes fall into four broad categories, all of which are lazily-loaded routes: each route imports its module on its first request, not at start-up. This ensures that start-up stays fast however many routes an app has.

LazilyLoadedRoute is an abstract base class. Its handler getter returns its lazilyLoadedHandler() method, bound to the route, and each subclass overrides lazilyLoadedHandler().

Post routes respond to POST requests and are identified by their .post.js and .post.ts extensions. They result in the creation of PostRoute instances.

A post route accepts uploads (acceptsUploads, from postRouteAcceptsUploads() in src/lib/uploadAcceptance.ts) when it requires sign-in or exports authorise. Its body parser stores the files in a multipart request only then (see Uploads).

Compared to other HTTP routes, a post route:

Like other HTTP routes, it then calls the route’s default export with { request, response } and responds with what it returns.

Socket routes simplify the use of WebSockets in Kitten apps and sites. They are identified by their .socket.js and .socket.ts extensions and result in the creation of WebSocketRoute instances. Unlike the paths of other routes, their paths include .socket (e.g., /chat.socket.js is at /chat.socket), as WebSocket clients don’t have to follow the redirect that adds a trailing slash.

Socket routes are exact GET routes, so echo.socket.js responds at /echo.socket and not at /echo.socket/extra/. They use the ws() method that the Kitten Server’s upgradeHandler() adds to WebSocket upgrade requests. A socket route accepts the connection, pings it every 30 seconds (terminating it if a ping is not answered with a pong), and adds it to the route’s list of connections, removing it when it closes. A request to a socket route that isn’t a WebSocket upgrade passes on to the next handler (next()), in case an HTTP route at the same path handles it.

A socket route that exports authorise fails to load, as only HTTP routes take it.

Kitten calls your route’s default export once per connection, with a parameter object that contains a request object as well as a socket object.

The socket object is a Proxy for the underlying ws WebSocket that adds two methods:

A message can be a string, a kitten.html result, or a Buffer. To send an object, pass it through JSON.stringify() first. Both methods return the number of connections they sent the message to. They, and the proxy’s send() method, also accept a promise (e.g., a kitten.html result with an async component). In that case, they send the message when the promise resolves, to the connections open at that time, and return a promise of the number of connections. Note that, unlike a page’s send() method, socket routes send plain strings as they are, without escaping them.

Kitten logs errors thrown by any of the following, and the server continues to run:

The Kitten Server also logs every WebSocket error event. To handle errors yourself, add your own error listener to the socket. Kitten closes any WebSocket, including a page’s, that sends a message over 1 MB (MAXIMUM_WEBSOCKET_MESSAGE_SIZE in src/Server.ts) with code 1009. A live page’s client (Kitten’s htmx WebSocket extension, in lib/ws.js) reconnects after code 1009, as it does after codes 1006, 1012, and 1013, and the page socket route then sends the page the reload command, as Kitten has removed it.

All other route types (e.g., .get.js, .head.js, etc.) are handled by the generic/simple HttpRoute class, which PostRoute extends.

On its first request, the HttpRoute.initialise() method imports the route’s module and checks its exports. The route fails to load if the onUploadLimit or authorise export isn’t a function, if the route requires sign-in and exports authorise, or if the uploadLimits export isn’t a plain object or contains an invalid limit or a key that isn’t a limit.

For each request, the HttpRoute.lazilyLoadedHandler() method then:

  1. If the route exports authorise, calls it with { request } before it reads the body. Unless it returns true, the route responds with 401 and the WWW-Authenticate: Bearer header. If it throws, the route logs the error with its file and responds with 500.

  2. Parses the request body with the route’s parser (createBodyParser() in src/lib/body.ts; see Uploads), and stops if the parser responded (e.g., with 400) or the client went away.

  3. Throws an error, which the error handler responds to with 500, when a route added through the router object parsed the body first and the route would have stored files (request.discardedFiles is above 0) or has its own uploadLimits export. The error’s message explains that middleware which passes requests on belongs in router.use.

  4. If the request exceeded an upload limit, throws an UploadLimitError, which Kitten responds to with 413, before calling the handler.

Otherwise, it calls the route’s default export with { request, response } and responds with the value that it returns (respondWith() in src/routes/lib/respondWith.ts):

A route that sets its own content type preserves it.

If the handler returns nothing (or another falsy value) and hasn’t started the response, Kitten ends the response with an empty body. If the handler returns any other value, Kitten throws an error, which the server’s error handler responds to with 500 and a message that reports what to return. If the handler has ended the response itself, Kitten does nothing else.

To see what the router and the socket routes are doing, turn on their log sections with the LOG environment variable, e.g.,

LOG=router,pageSocketRoute,webSocketRoute,lifecycle,event,streamingHtml kitten

Source code:

The Kitten Server never adds a route’s handler to Polka directly. It adds the handler that the routeHandler() function in src/routes/lib/routeHandler.ts returns, for Kitten’s routes, the app’s routes, /uploads/:id/:download?, the __archive__ folders, and the routes added through the router object. (The handlerOfRoute() function creates it for a route from a file.) The route handler decides only for its own route, never for another route that might match the request, and it runs these steps in this order:

  1. On a route that doesn’t respond to WebSocket upgrades (every route apart from socket routes and page socket routes), passes an upgrade on (next()).

  2. On a socket route or a page socket route, passes on every request that isn’t an upgrade.

  3. If the route’s file path requires sign-in (requiresSignIn()) and the session isn’t signed in, refuses the request with requireSignIn() from src/lib/signIn.ts: it responds with 303 See Other to /šŸ’•/sign-in/. For a GET request that isn’t an upgrade, the 303 includes the requested URL in the sign-in page’s return parameter (unless the URL is over 2,048 characters, RETURN_PATH_LENGTH_LIMIT). It responds with 403 Forbidden when the request has no session. Socket routes reach this step only for upgrades, which are refused with 303 too.

  4. Runs the route’s handler.

The handler that the routeHandler() function returns has the route’s file path as its filePath property, which you can see when you list a route’s handlers in the interactive shell.

The downloadHandler() function in src/Uploads.ts uses the same requireSignIn() function for private uploads (see Uploads), so there is one copy of the sign-in rule.

Source code:

The routerFor() function in src/lib/router.ts creates the router object that the Kitten Server passes to main.script.js and later sets as kitten.app.router. It is an object whose prototype is the Polka app, so use(), find(), routes, and every other property resolve to the app. Its route methods (get, head, post, put, patch, delete, options, trace, all, and add) add each handler through the route handler, so the routes pass upgrades on. The use() method is the app’s own, so middleware added with it runs for every request that reaches it.

Routes for post, put, patch, delete, and all (and add with one of those methods, or with every method) parse the request body before their handlers run, with a parser that stores no files. A request over a limit throws an UploadLimitError, which the error handler responds to with 413.

The routerFor() function records the method and pattern of each route added with a string pattern. Once all routes are added, the Kitten Server refuses a route added through the router object at the same method and normalised pattern as an app route, or under /šŸ’•/ or /🐱/, or below /uploads/ (see Initialisation (asynchronous)).

Middleware added with router.use runs:

Middleware reads headers; routes read bodies.

A router handler for a method with a body that calls next() (e.g., router.all('*', …) that logs requests) parses the body before the file route behind it runs, without storing files. kitten-busboy records the number of files it discarded in request.discardedFiles. A file route that would have stored files, or that has its own uploadLimits export, then responds with 500 (see Other HTTP routes), as its files are lost and its limits are not applied. Middleware like this belongs in router.use.

Source code:

The Files class finds the files in its base path, sorts them by extension category type, and watches the base path for changes, dispatching file events when changes occur.

It is used by Routes when creating the file system-based routes for Kitten Server. Kitten Server creates routes twice, so there are two Files instances, each with its own watcher: one for your app (process.env.basePath) and one for Kitten’s own web app (paths.KITTEN_WEB_APP_DIRECTORY_PATH).

The Files class is an EventTarget.

The Files class is instantiated using an asynchronous factory method (new). The base path defaults to process.env.basePath. Calling the constructor or initialise() directly throws an error.

const files = await Files.new(basePath)

The Files.new() method checks the base path with refuseUnservablePath() and crawls it with routedFilePaths(), both of which the kitten check command also calls. It rejects with:

The Routes instance reports any of these errors and exits with systemExitCodes.dataError (65). Since this is not a restart request, the Kitten Process Manager quits instead of starting a new Kitten process that would fail in the same way.

Files are categorised by extension category type:

console.log(files.byExtensionCategoryType)

Files uses the Watcher module for file system watching and the tiny-readdir module to get the list of files from the file system. (The latter module is also used by Watcher itself.)

While compiling the list of paths for routing, Kitten tests each path, relative to the base path, against ignoreForRoutingRegExp and ignores:

The watcher uses ignoreForWatchingRegExp, which ignores only hidden paths (with the same exceptions) and paths that contain node_modules. So Kitten watches app_modules and __archive__ folders for changes. This results in live reloads during development if you change a file in the app module you’re working on (just like with any other file in your Kitten app/site).

Currently, the file map is created twice.

First, Kitten uses tiny-readdir (which is also used by Watcher), to create its initial list of files. Then it instantiates Watcher. Kitten should be able to pass the returned readdirMap to Watcher so it doesn’t have to replicate the directory traversal but there is a bug in Watcher that’s currently preventing this.

Keep an eye on the following two Watcher issues:

We’re tracking this in issue #366.

To catch Kitten being run in the wrong folder (e.g., a folder that contains many projects) before it runs out of memory, the initial crawl and the watcher both have limits:

To raise a limit, set the environment variable of the same name with a KITTEN_ prefix (KITTEN_MAXIMUM_NUMBER_OF_PATHS or KITTEN_MAXIMUM_DIRECTORY_DEPTH) when you run Kitten.

Files dispatches a file event (a CustomEvent) whose detail is a FileEventDetails instance with itemType, eventType, and itemPath properties. It dispatches one for each of the following watcher events (shown with their itemType and eventType):

The addFile() method also dispatches a file event with an itemType of route and an eventType of added for each category it adds a file to. It does this during the initial crawl, before Routes adds its listener, and for each file the watcher adds.

Kitten logs each watcher event in the watcher log section (e.g., LOG=watcher kitten).

On a watcher error, Kitten logs the error and closes the watcher. Files dispatches no more events after this, so file changes don’t restart the server until Kitten restarts for another reason. (files.close() also closes the watcher.)

The Routes instance listens for file events only in development (when process.env.PRODUCTION is not set). On any file event, the handleFileChange() method logs the change (e.g., ā€œšŸ”” File changed (/index.page.js), exiting to trigger reload.ā€) and exits with systemExitCodes.restartRequest (99). The Kitten Process Manager then starts a new Kitten process.

Flowchart: From a file change to a reloaded page

Kitten restarts because Node can’t unload an ES module, and each route imports its module only once. A restart also creates all routes again and runs your main.script and database app module again.

In development, pages (.page.js, .page.ts, and .page.md), static HTML files (.html and .htm), and Kitten’s error pages include developmentTimeHotReloadScript. This loads kitten-hot-reload-1.js, which connects to the development socket at /🐱/development-socket/. The websocket middleware sends the boot ID (bootId), which it creates once per process, on each connection. The script tag includes the boot ID of the process that rendered the page (data-kitten-boot-id), and the script reloads the page when a boot ID it receives differs from it. This ensures that the page reloads after a restart, including a restart between the render and the socket’s first connection. The script connects again 50 milliseconds after the socket closes. It also closes the socket when the browser hides the page (the pagehide event), as pages with open sockets are not eligible for the browser’s back/forward cache, and connects again when the browser restores the page from that cache (the pageshow event). (In production, developmentTimeHotReloadScript is empty.)

In production, nothing listens for file events, so changing a file doesn’t restart the server. (Files still creates a watcher.) A production server restarts when Kitten exits with any code other than 0 (e.g., with a restart request after an update, for its daily maintenance restart, or on an uncaught exception) and systemd starts it again (see Errors and restarts).

In development, any write in the project folder outside the ignored paths restarts the server and reloads all open pages. This includes writes to files Kitten never routes, loads, or serves (e.g., Playwright’s test-results/ folder). We’re tracking this in issue #346.

Files are sorted into the following category types, based on their extensions. A file’s extension is its last two dot-separated parts (extensionOfFilePath()), so index.page.js has the extension page.js and styles.css has the extension css. Each category maps an extension to the paths of the files with that extension (e.g., files.byExtensionCategoryType.frontendRoutes['page.js']), and a file can be in more than one category (e.g., a page is in frontendRoutes, dynamicRoutes, and allRoutes). The extension lists are in Utils.ts.

Kitten uses only the staticRoutes and dynamicRoutes categories (in the routeFilesFor() function in src/lib/routeFiles.ts, which the Routes class and the kitten check command call). Nothing else in Kitten reads the backendRoutes, frontendRoutes, dependencies, or allRoutes categories of byExtensionCategoryType. (Note that, despite its name, allRoutes doesn’t contain static routes.)

Since every file that isn’t a dynamic route or a dependency is a static route, Kitten currently serves server-only .js and .ts helpers (issue #344) as well as project files that really shouldn’t be served, e.g., editor backups like index.page.js~, an accidentally copied-in key, package.json (issue #363), etc.

Source code:

Polka passes errors thrown by middleware and routes, as well as requests that no route handles (other than WebSocket upgrades, which the noMatchHandler() method responds to itself), to the default export of src/ServerError.ts. It responds with:

If the route has already started its response, the error handler cannot show an error page, so it logs the error and destroys the response (if the response hasn’t ended). The error handler also counts missing pages and server errors in the missing and serverErrors stats.

Kitten exits with these codes:

In development, the Kitten Process Manager starts a new worker for code 99 and quits for any other code. In production, the systemd service unit that the kitten deploy command writes has Restart=on-failure, so systemd restarts Kitten a second after any exit other than a clean one (a code other than 0, or a signal, e.g., the start deadline’s SIGKILL).

Kitten does not limit the number of restarts. So, in development, an uncaught exception at every start restarts Kitten in a loop.

Source code:

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.