# Kubo changelog v0.33

- [v0.33.0](#v0310)

## v0.33.0

- [Overview](#overview)
- [🔦 Highlights](#-highlights)
  - [Shared TCP listeners](#shared-tcp-listeners)
  - [AutoTLS takes care of Secure WebSockets setup](#autotls-takes-care-of-secure-websockets-setup)
  - [Bitswap improvements from Boxo](#bitswap-improvements-from-boxo)
  - [Using default `libp2p_rcmgr`  metrics](#using-default-libp2p_rcmgr--metrics)
  - [Flatfs does not `sync` on each write](#flatfs-does-not-sync-on-each-write)
  - [`ipfs add --to-files` no longer works with `--wrap`](#ipfs-add---to-files-no-longer-works-with---wrap)
  - [`ipfs --api` supports HTTPS RPC endpoints](#ipfs---api-supports-https-rpc-endpoints)
  - [New options for faster writes: `WriteThrough`, `BlockKeyCacheSize`, `BatchMaxNodes`, `BatchMaxSize`](#new-options-for-faster-writes-writethrough-blockkeycachesize-batchmaxnodes-batchmaxsize)
  - [MFS stability with large number of writes](#mfs-stability-with-large-number-of-writes)
  - [New DoH resolvers for non-ICANN DNSLinks](#new-doh-resolvers-for-non-icann-dnslinks)
  - [📦️ Important dependency updates](#-important-dependency-updates)
  - [Escape Redirect URL for Directory](#escape-redirect-url-for-directory)
- [📝 Changelog](#-changelog)
- [👨‍👩‍👧‍👦 Contributors](#-contributors)

### Overview

### 🔦 Highlights

#### Shared TCP listeners

Kubo now supports sharing the same TCP port (`4001` by default) by both [raw TCP](https://github.com/ipfs/kubo/blob/master/docs/config.md#swarmtransportsnetworktcp) and [WebSockets](https://github.com/ipfs/kubo/blob/master/docs/config.md#swarmtransportsnetworkwebsocket)  libp2p transports.

This feature is not yet compatible with Private Networks and can be disabled by setting `LIBP2P_TCP_MUX=false` if causes any issues.

#### AutoTLS takes care of Secure WebSockets setup

It is no longer necessary to manually add `/tcp/../ws` listeners to `Addresses.Swarm` when [`AutoTLS.Enabled`](https://github.com/ipfs/kubo/blob/master/docs/config.md#autotlsenabled) is set to `true`. Kubo will detect if `/ws` listener is missing and add one on the same port as pre-existing TCP (e.g. `/tcp/4001`), removing the need for any extra configuration.
> [!TIP]
> Give it a try:
> ```console
> $ ipfs config --json AutoTLS.Enabled true
> ```
> And restart the node. If you are behind NAT,  make sure your node is publicly diallable (uPnP or port forwarding), and wait a few minutes to pass all checks and for the changes to take effect.

See [`AutoTLS`](https://github.com/ipfs/kubo/blob/master/docs/config.md#autotls) for more information.

#### Bitswap improvements from Boxo

This release includes some refactorings and improvements affecting Bitswap which should improve reliability. One of the changes affects blocks providing. Previously, the bitswap layer took care itself of announcing new blocks -added or received- with the configured provider (i.e. DHT). This bypassed the "Reprovider", that is, the system that manages precisely "providing" the blocks stored by Kubo. The Reprovider knows how to take advantage of the [AcceleratedDHTClient](https://github.com/ipfs/kubo/blob/master/docs/config.md#routingaccelerateddhtclient), is able to handle priorities, logs statistics and is able to resume on daemon reboot where it left off. From now on, Bitswap will not be doing any providing on-the-side and all announcements are managed by the reprovider. In some cases, when the reproviding queue is full with other elements, this may cause additional delays, but more likely this will result in improved block-providing behaviour overall.

#### Using default `libp2p_rcmgr`  metrics

Bespoke rcmgr metrics [were removed](https://github.com/ipfs/kubo/pull/9947), Kubo now exposes only the default `libp2p_rcmgr` metrics from go-libp2p.
This makes it easier to compare Kubo with custom implementations based on go-libp2p.
If you depended on removed ones, please fill an issue to add them to the upstream [go-libp2p](https://github.com/libp2p/go-libp2p).

#### Flatfs does not `sync` on each write

New repositories initialized with `flatfs` in `Datastore.Spec` will have `sync` set to `false`.

The old default was overly conservative and caused performance issues in big repositories that did a lot of writes. There is usually no need to flush on every block write to disk before continuing. Setting this to false is safe as kubo will automatically flush writes to disk before and after performing critical operations like pinning. However, we still provide users with ability to set this to true to be extra-safe (at the cost of a slowdown when adding files in bulk).

#### `ipfs add --to-files` no longer works with `--wrap`

Onboarding files and directories with `ipfs add --to-files` now requires non-empty names. due to this, The `--to-files` and `--wrap` options are now mutually exclusive ([#10612](https://github.com/ipfs/kubo/issues/10612)).

#### `ipfs --api` supports HTTPS RPC endpoints

CLI and RPC client now supports accessing Kubo RPC over `https://` protocol when multiaddr ending with `/https` or `/tls/http` is passed to `ipfs --api`:

```console
$ ipfs id --api /dns/kubo-rpc.example.net/tcp/5001/tls/http
# → https://kubo-rpc.example.net:5001
```

#### New options for faster writes: `WriteThrough`, `BlockKeyCacheSize`, `BatchMaxNodes`, `BatchMaxSize`

Now that Kubo supports [`pebble`](https://github.com/ipfs/kubo/blob/master/docs/datastores.md#pebbleds) as a datastore backend, it becomes very useful to expose some additional configuration options for how the blockservice/blockstore/datastore combo behaves.

Usually, LSM-tree based datastore like Pebble or Badger have very fast write performance (blocks are streamed to disk) while incurring in read-amplification penalties (blocks need to be looked up in the index to know where they are on disk),  specially noticiable on spinning disks.

Prior to this version, `BlockService` and `Blockstore` implementations performed a `Has(cid)` for every block that was going to be written, skipping the writes altogether if the block was already present in the datastore. The performance impact of this `Has()` call can vary. The `Datastore` implementation itself might include block-caching and things like bloom-filters to speed up lookups and mitigate read-penalties. Our `Blockstore` implementation also supports a bloom-filter (controlled by `BloomFilterSize` and disabled by default), and a two-queue cache for keys and block sizes. If we assume that most of the blocks added to Kubo are new blocks, not already present in the datastore, or that the datastore itself includes mechanisms to optimize writes and avoid writing the same data twice, the calls to `Has()` at both BlockService and Blockstore layers seem superflous to they point they even harm write performance.

For these reasons, from now on, the default is to use a "write-through" mode for the Blockservice and the Blockstore. We have added a new option `Datastore.WriteThrough`, which defaults to `true`. Previous behaviour can be obtained by manually setting it to `false`.

We have also made the size of the two-queue blockstore cache configurable with another option: `Datastore.BlockKeyCacheSize`, which defaults to `65536` (64KiB). Additionally, this caching layer can be disabled altogether by setting it to `0`. In particular, this option controls the size of a blockstore caching layer that records whether the blockstore has certain block and their sizes (but does not cache the contents, so it stays relativey small in general).

Finally, we have added two new options to the `Import` section to control the maximum size of write-batches: `BatchMaxNodes` and `BatchMaxSize`. These are set by default to `128` nodes and `20MiB`. Increasing them will batch more items together when importing data with `ipfs dag import`, which can speed things up. It is importance to find a balance between available memory (used to hold the batch), disk latencies (when writing the batch) and processing power (when preparing the batch, as nodes are sorted and duplicates removed).

As a reminder, details from all the options are explained in the [configuration documentation](https://github.com/ipfs/kubo/blob/master/docs/config.md).

We recommend users trying Pebble as a datastore backend to disable both blockstore bloom-filter and key caching layers and enable write through as a way to evaluate the raw performance of the underlying datastore, which includes its own bloom-filter and caching layers (default cache size is `8MiB` and can be configured in the [options](https://github.com/ipfs/kubo/blob/master/docs/datastores.md#pebbleds).

#### MFS stability with large number of writes

We have fixed a number of issues that were triggered by writing or copying many files onto an MFS folder: increased memory usage first, then CPU, disk usage, and eventually a deadlock on write operations. The details of the fixes can be read at [#10630](https://github.com/ipfs/kubo/pull/10630) and [#10623](https://github.com/ipfs/kubo/pull/10623). The result is that writing large amounts of files to an MFS folder should now be possible without major issues. It is possible, as before, to speed up the operations using the `ipfs files --flush=false <op> ...` flag, but it is recommended to switch to `ipfs files --flush=true <op> ...` regularly, or call `ipfs files flush` on the working directory regularly, as this will flush, clear the directory cache and speed up reads. 

#### New DoH resolvers for non-ICANN DNSLinks

- `.eth` TLD DNSLinks are now resolved via [DNS-over-HTTPS](https://en.wikipedia.org/wiki/DNS_over_HTTPS) endpoint at `https://dns.eth.limo/dns-query`
- `.crypto` TLD DNSLinks are now resolved via DoH endpoint at `https://resolver.unstoppable.io/dns-query`

#### 📦️ Important dependency updates

- update `boxo` to [v0.27.2](https://github.com/ipfs/boxo/releases/tag/v0.27.2) (incl. [v0.25.0](https://github.com/ipfs/boxo/releases/tag/v0.25.0) + [v0.26.0](https://github.com/ipfs/boxo/releases/tag/v0.26.0) + [v0.27.0](https://github.com/ipfs/boxo/releases/tag/v0.27.0) + [v0.27.1](https://github.com/ipfs/boxo/releases/tag/v0.27.1))
- update `go-libp2p` to [v0.38.2](https://github.com/libp2p/go-libp2p/releases/tag/v0.38.2) (incl. [v0.37.1](https://github.com/libp2p/go-libp2p/releases/tag/v0.37.1) + [v0.37.2](https://github.com/libp2p/go-libp2p/releases/tag/v0.37.2) + [v0.38.0](https://github.com/libp2p/go-libp2p/releases/tag/v0.38.0) + [v0.38.1](https://github.com/libp2p/go-libp2p/releases/tag/v0.38.1))
- update `go-libp2p-kad-dht` to [v0.28.2](https://github.com/libp2p/go-libp2p-kad-dht/releases/tag/v0.28.2)
- update `quic-go` to [v0.49.0](https://github.com/quic-go/quic-go/releases/tag/v0.49.0)
- update `p2p-forge/client` to [v0.3.0](https://github.com/ipshipyard/p2p-forge/releases/tag/v0.3.0) (incl. [v0.1.0](https://github.com/ipshipyard/p2p-forge/releases/tag/v0.1.0), [v0.2.0](https://github.com/ipshipyard/p2p-forge/releases/tag/v0.2.0), [v0.2.1](https://github.com/ipshipyard/p2p-forge/releases/tag/v0.2.1), [v0.2.2](https://github.com/ipshipyard/p2p-forge/releases/tag/v0.2.2))
- update `ipfs-webui` to [v4.4.2](https://github.com/ipfs/ipfs-webui/releases/tag/v4.4.2) (incl. [v4.4.1](https://github.com/ipfs/ipfs-webui/releases/tag/v4.4.1))

#### Escape Redirect URL for Directory

When navigating to a subdirectory, served by the Kubo web server, a subdirectory without a trailing slash gets redirected to a URL with a trailing slash. If there are special characters such as "%" in the subdirectory name then these must be escaped in the redirect URL. Previously this was not being done and was preventing navigation to such subdirectories, requiring the user to manually add a trailing slash to the subdirectory URL. This is now fixed to handle the redirect to URLs with characters that must be escaped.

### 📝 Changelog

### 👨‍👩‍👧‍👦 Contributors
