Merge branch 'master' into master

2025-06-27 07:57:30 +08:00 · 2018-08-23 07:53:58 +01:00
parent 31711e165f 4fb2666a24
commit 1f6cb89e56
37 changed files with 490 additions and 292 deletions
--- a/README.md
+++ b/README.md
@ -9,8 +9,6 @@
 [![GoDoc](https://godoc.org/github.com/ipfs/go-ipfs?status.svg)](https://godoc.org/github.com/ipfs/go-ipfs)
 [![Build Status](https://travis-ci.org/ipfs/go-ipfs.svg?branch=master)](https://travis-ci.org/ipfs/go-ipfs)

-> IPFS implementation in Go
-
 ## Project Status

 [![Throughput Graph](https://graphs.waffle.io/ipfs/go-ipfs/throughput.svg)](https://waffle.io/ipfs/go-ipfs/metrics/throughput)
@ -39,16 +37,15 @@ Please put all issues regarding:
    - [Download and Compile IPFS](#download-and-compile-ipfs)
    - [Troubleshooting](#troubleshooting)
  - [Development Dependencies](#development-dependencies)
-  - [Updating](#updating)
+  - [Updating](#updating-go-ipfs)
 - [Usage](#usage)
 - [Getting Started](#getting-started)
  - [Some things to try](#some-things-to-try)
  - [Docker usage](#docker-usage)
  - [Troubleshooting](#troubleshooting-1)
 - [Packages](#packages)
+- [Development](#development)
 - [Contributing](#contributing)
-  - [Want to hack on IPFS?](#want-to-hack-on-ipfs)
-  - [Want to read our code?](#want-to-read-our-code)
 - [License](#license)

 ## Security Issues
@ -89,13 +86,15 @@ You can also download go-ipfs from this project's GitHub releases page if you ar
 In Arch Linux go-ipfs is available as
 [go-ipfs](https://www.archlinux.org/packages/community/x86_64/go-ipfs/) package.

-	$ sudo pacman -S go-ipfs
+```
+$ sudo pacman -S go-ipfs
+```

 Development version of go-ipfs is also on AUR under
 [go-ipfs-git](https://aur.archlinux.org/packages/go-ipfs-git/).
 You can install it using your favourite AUR Helper or manually from AUR.

-### Nix
+#### Nix

 For Linux and MacOSX you can use the purely functional package manager [Nix](https://nixos.org/nix/):

@ -108,7 +107,9 @@ You can also install the Package by using it's attribute name, which is also `ip

 With snap, in any of the [supported Linux distributions](https://snapcraft.io/docs/core/install):

-    $ sudo snap install ipfs
+```
+$ sudo snap install ipfs
+```

 ### Build from Source

@ -116,7 +117,6 @@ With snap, in any of the [supported Linux distributions](https://snapcraft.io/do

 The build process for ipfs requires Go 1.10 or higher. If you don't have it: [Download Go 1.10+](https://golang.org/dl/).

-
 You'll need to add Go's bin directories to your `$PATH` environment variable e.g., by adding these lines to your `/etc/profile` (for a system-wide installation) or `$HOME/.profile`:

 ```
@ -154,43 +154,44 @@ mismatched APIs.

 #### Troubleshooting

-* Separate [instructions are available for building on Windows](docs/windows.md).
-* Also, [instructions for OpenBSD](docs/openbsd.md).
-* `git` is required in order for `go get` to fetch all dependencies.
-* Package managers often contain out-of-date `golang` packages.
+- Separate [instructions are available for building on Windows](docs/windows.md).
+- Also, [instructions for OpenBSD](docs/openbsd.md).
+- `git` is required in order for `go get` to fetch all dependencies.
+- Package managers often contain out-of-date `golang` packages.
  Ensure that `go version` reports at least 1.10. See above for how to install go.
-* If you are interested in development, please install the development
+- If you are interested in development, please install the development
 dependencies as well.
-* *WARNING: Older versions of OSX FUSE (for Mac OS X) can cause kernel panics when mounting!*
+- _WARNING_: Older versions of OSX FUSE (for Mac OS X) can cause kernel panics when mounting!-
  We strongly recommend you use the [latest version of OSX FUSE](http://osxfuse.github.io/).
  (See https://github.com/ipfs/go-ipfs/issues/177)
-* For more details on setting up FUSE (so that you can mount the filesystem), see the docs folder.
-* Shell command completion is available in `misc/completion/ipfs-completion.bash`. Read [docs/command-completion.md](docs/command-completion.md) to learn how to install it.
-* See the [init examples](https://github.com/ipfs/website/tree/master/static/docs/examples/init) for how to connect IPFS to systemd or whatever init system your distro uses.
+- For more details on setting up FUSE (so that you can mount the filesystem), see the docs folder.
+- Shell command completion is available in `misc/completion/ipfs-completion.bash`. Read [docs/command-completion.md](docs/command-completion.md) to learn how to install it.
+- See the [init examples](https://github.com/ipfs/website/tree/master/static/docs/examples/init) for how to connect IPFS to systemd or whatever init system your distro uses.

-### Development Dependencies
+### Updating go-ipfs

-If you make changes to the protocol buffers, you will need to install the [protoc compiler](https://github.com/google/protobuf).
+#### Using ipfs-update

-### Updating
-
-#### Updating using ipfs-update
 IPFS has an updating tool that can be accessed through `ipfs update`. The tool is
 not installed alongside IPFS in order to keep that logic independent of the main
 codebase. To install `ipfs update`, [download it here](https://ipfs.io/ipns/dist.ipfs.io/#ipfs-update).

 #### Downloading IPFS builds using IPFS
+
 List the available versions of go-ipfs:
+
 ```
 $ ipfs cat /ipns/dist.ipfs.io/go-ipfs/versions
 ```

 Then, to view available builds for a version from the previous command ($VERSION):
+
 ```
 $ ipfs ls /ipns/dist.ipfs.io/go-ipfs/$VERSION
 ```

 To download a given build of a version:
+
 ```
 $ ipfs get /ipns/dist.ipfs.io/go-ipfs/$VERSION/go-ipfs_$VERSION_darwin-386.tar.gz # darwin 32-bit build
 $ ipfs get /ipns/dist.ipfs.io/go-ipfs/$VERSION/go-ipfs_$VERSION_darwin-amd64.tar.gz # darwin 64-bit build
@ -318,18 +319,11 @@ Stop the running container:

 ### Troubleshooting

-If you have previously installed IPFS before and you are running into
-problems getting a newer version to work, try deleting (or backing up somewhere
-else) your IPFS config directory (~/.ipfs by default) and rerunning `ipfs init`.
-This will reinitialize the config file to its defaults and clear out the local
-datastore of any bad entries.
+If you have previously installed IPFS before and you are running into problems getting a newer version to work, try deleting (or backing up somewhere else) your IPFS config directory (~/.ipfs by default) and rerunning `ipfs init`. This will reinitialize the config file to its defaults and clear out the local datastore of any bad entries.

-Please direct general questions and help requests to our
-[forum](https://discuss.ipfs.io) or our IRC channel (freenode #ipfs).
+Please direct general questions and help requests to our [forum](https://discuss.ipfs.io) or our IRC channel (freenode #ipfs).

-If you believe you've found a bug, check the [issues list](https://github.com/ipfs/go-ipfs/issues)
-and, if you don't see your problem there, either come talk to us on IRC (freenode #ipfs) or
-file an issue of your own!
+If you believe you've found a bug, check the [issues list](https://github.com/ipfs/go-ipfs/issues) and, if you don't see your problem there, either come talk to us on IRC (freenode #ipfs) or file an issue of your own!

 ## Packages

@ -399,28 +393,48 @@ Listing of the main packages used in the IPFS ecosystem. There are also three sp
 | [`go-ipfs-chunker`](//github.com/ipfs/go-ipfs-chunker) | [![Travis CI](https://travis-ci.org/ipfs/go-ipfs-chunker.svg?branch=master)](https://travis-ci.org/ipfs/go-ipfs-chunker) | N/A | [![codecov](https://codecov.io/gh/ipfs/go-ipfs-chunker/branch/master/graph/badge.svg)](https://codecov.io/gh/ipfs/go-ipfs-chunker) |
 | [`go-ipfs-ds-help`](//github.com/ipfs/go-ipfs-ds-help) | [![Travis CI](https://travis-ci.org/ipfs/go-ipfs-ds-help.svg?branch=master)](https://travis-ci.org/ipfs/go-ipfs-ds-help) | N/A | [![codecov](https://codecov.io/gh/ipfs/go-ipfs-ds-help/branch/master/graph/badge.svg)](https://codecov.io/gh/ipfs/go-ipfs-ds-help) |

+## Development
+
+Some places to get you started on the codebase:
+
+- Main file: [./cmd/ipfs/main.go](https://github.com/ipfs/go-ipfs/blob/master/cmd/ipfs/main.go)
+- CLI Commands: [./core/commands/](https://github.com/ipfs/go-ipfs/tree/master/core/commands)
+- Bitswap (the data trading engine): [go-bitswap](https://github.com/ipfs/go-bitswap)
+- libp2p
+  - libp2p: https://github.com/libp2p/go-libp2p
+  - DHT: https://github.com/libp2p/go-libp2p-kad-dht
+  - PubSub: https://github.com/libp2p/go-floodsub
+
+### CLI, HTTP-API, Architecture Diagram
+
+![](./docs/cli-http-api-core-diagram.png)
+
+> [Origin](https://github.com/ipfs/pm/pull/678#discussion_r210410924)
+
+Description: Dotted means "likely going away". The "Legacy" parts are thin wrappers around some commands to translate between the new system and the old system. The grayed-out parts on the "daemon" diagram are there to show that the code is all the same, it's just that we turn some pieces on and some pieces off depending on whether we're running on the client or the server.
+
+### Testing
+
+```
+make test
+```
+
+### Development Dependencies
+
+If you make changes to the protocol buffers, you will need to install the [protoc compiler](https://github.com/google/protobuf).
+
+### Developer Notes
+
+Find more documentation for developers on [docs](./docs)
+
 ## Contributing

+[![](https://cdn.rawgit.com/jbenet/contribute-ipfs-gif/master/img/contribute.gif)](https://github.com/ipfs/community/blob/master/contributing.md)
+
 We ❤️ all [our contributors](docs/AUTHORS); this project wouldn’t be what it is without you! If you want to help out, please see [Contribute.md](contribute.md).

 This repository falls under the IPFS [Code of Conduct](https://github.com/ipfs/community/blob/master/code-of-conduct.md).

-### Want to hack on IPFS?
-
-[![](https://cdn.rawgit.com/jbenet/contribute-ipfs-gif/master/img/contribute.gif)](https://github.com/ipfs/community/blob/master/contributing.md)
-
-### Want to read our code?
-
-Some places to get you started. (WIP)
-
-Main file: [cmd/ipfs/main.go](https://github.com/ipfs/go-ipfs/blob/master/cmd/ipfs/main.go) <br>
-CLI Commands: [core/commands/](https://github.com/ipfs/go-ipfs/tree/master/core/commands) <br>
-Bitswap (the data trading engine): [exchange/bitswap/](https://github.com/ipfs/go-ipfs/tree/master/exchange/bitswap)
-
-DHT: https://github.com/libp2p/go-libp2p-kad-dht <br>
-PubSub: https://github.com/libp2p/go-floodsub <br>
-libp2p: https://github.com/libp2p/go-libp2p
-
 ## License

-MIT
+[MIT](./LICENSE)
--- a/cmd/ipfs/dnsresolve_test.go
+++ b/cmd/ipfs/dnsresolve_test.go
@ -0,0 +1,72 @@
+package main
+
+import (
+	"context"
+	"fmt"
+	"net"
+	"strings"
+	"testing"
+
+	ma "gx/ipfs/QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7/go-multiaddr"
+	madns "gx/ipfs/QmfXU2MhWoegxHoeMd3A2ytL2P6CY4FfqGWc23LTNWBwZt/go-multiaddr-dns"
+)
+
+var (
+	ctx         = context.Background()
+	testAddr, _ = ma.NewMultiaddr("/dns4/example.com/tcp/5001")
+)
+
+func makeResolver(n uint8) *madns.Resolver {
+	results := make([]net.IPAddr, n)
+	for i := uint8(0); i < n; i++ {
+		results[i] = net.IPAddr{IP: net.ParseIP(fmt.Sprintf("192.0.2.%d", i))}
+	}
+
+	backend := &madns.MockBackend{
+		IP: map[string][]net.IPAddr{
+			"example.com": results,
+		}}
+
+	return &madns.Resolver{
+		Backend: backend,
+	}
+}
+
+func TestApiEndpointResolveDNSOneResult(t *testing.T) {
+	dnsResolver = makeResolver(1)
+
+	addr, err := resolveAddr(ctx, testAddr)
+	if err != nil {
+		t.Error(err)
+	}
+
+	if ref, _ := ma.NewMultiaddr("/ip4/192.0.2.0/tcp/5001"); !addr.Equal(ref) {
+		t.Errorf("resolved address was different than expected")
+	}
+}
+
+func TestApiEndpointResolveDNSMultipleResults(t *testing.T) {
+	dnsResolver = makeResolver(4)
+
+	addr, err := resolveAddr(ctx, testAddr)
+	if err != nil {
+		t.Error(err)
+	}
+
+	if ref, _ := ma.NewMultiaddr("/ip4/192.0.2.0/tcp/5001"); !addr.Equal(ref) {
+		t.Errorf("resolved address was different than expected")
+	}
+}
+
+func TestApiEndpointResolveDNSNoResults(t *testing.T) {
+	dnsResolver = makeResolver(0)
+
+	addr, err := resolveAddr(ctx, testAddr)
+	if addr != nil || err == nil {
+		t.Error("expected test address not to resolve, and to throw an error")
+	}
+
+	if !strings.HasPrefix(err.Error(), "non-resolvable API endpoint") {
+		t.Errorf("expected error not thrown; actual: %v", err)
+	}
+}
--- a/cmd/ipfs/main.go
+++ b/cmd/ipfs/main.go
@ -34,6 +34,7 @@ import (
 	osh "gx/ipfs/QmXuBJ7DR6k3rmUEKtvVMhwjmXDuJgXXPUt4LQXKBMsU93/go-os-helper"
 	ma "gx/ipfs/QmYmsdtJ3HsodkePE3eU3TsCaP2YvPZJ4LoXnNkDE5Tpt7/go-multiaddr"
 	loggables "gx/ipfs/QmZ4zF1mBrt8C2mSCM4ZYE4aAnv78f7GvrzufJC4G5tecK/go-libp2p-loggables"
+	madns "gx/ipfs/QmfXU2MhWoegxHoeMd3A2ytL2P6CY4FfqGWc23LTNWBwZt/go-multiaddr-dns"
 )

 // log is the command logger
@ -41,6 +42,9 @@ var log = logging.Logger("cmd/ipfs")

 var errRequestCanceled = errors.New("request canceled")

+// declared as a var for testing purposes
+var dnsResolver = madns.DefaultResolver
+
 const (
 	EnvEnableProfiling = "IPFS_PROF"
 	cpuProfile         = "ipfs.cpuprof"
@ -235,7 +239,7 @@ func commandShouldRunOnDaemon(details cmdDetails, req *cmds.Request, cctx *oldcm
 	// did user specify an api to use for this command?
 	apiAddrStr, _ := req.Options[corecmds.ApiOption].(string)

-	client, err := getApiClient(cctx.ConfigRoot, apiAddrStr)
+	client, err := getAPIClient(req.Context, cctx.ConfigRoot, apiAddrStr)
 	if err == repo.ErrApiNotRunning {
 		if apiAddrStr != "" && req.Command != daemonCmd {
 			// if user SPECIFIED an api, and this cmd is not daemon
@ -403,10 +407,10 @@ If you're sure go-ipfs isn't running, you can just delete it.
 var checkIPFSUnixFmt = "Otherwise check:\n\tps aux | grep ipfs"
 var checkIPFSWinFmt = "Otherwise check:\n\ttasklist | findstr ipfs"

-// getApiClient checks the repo, and the given options, checking for
+// getAPIClient checks the repo, and the given options, checking for
 // a running API service. if there is one, it returns a client.
 // otherwise, it returns errApiNotRunning, or another error.
-func getApiClient(repoPath, apiAddrStr string) (http.Client, error) {
+func getAPIClient(ctx context.Context, repoPath, apiAddrStr string) (http.Client, error) {
 	var apiErrorFmt string
 	switch {
 	case osh.IsUnix():
@ -440,10 +444,15 @@ func getApiClient(repoPath, apiAddrStr string) (http.Client, error) {
 	if len(addr.Protocols()) == 0 {
 		return nil, fmt.Errorf(apiErrorFmt, repoPath, "multiaddr doesn't provide any protocols")
 	}
-	return apiClientForAddr(addr)
+	return apiClientForAddr(ctx, addr)
 }

-func apiClientForAddr(addr ma.Multiaddr) (http.Client, error) {
+func apiClientForAddr(ctx context.Context, addr ma.Multiaddr) (http.Client, error) {
+	addr, err := resolveAddr(ctx, addr)
+	if err != nil {
+		return nil, err
+	}
+
 	_, host, err := manet.DialArgs(addr)
 	if err != nil {
 		return nil, err
@ -451,3 +460,19 @@ func apiClientForAddr(addr ma.Multiaddr) (http.Client, error) {

 	return http.NewClient(host, http.ClientWithAPIPrefix(corehttp.APIPath)), nil
 }
+
+func resolveAddr(ctx context.Context, addr ma.Multiaddr) (ma.Multiaddr, error) {
+	ctx, cancelFunc := context.WithTimeout(ctx, 10*time.Second)
+	defer cancelFunc()
+
+	addrs, err := dnsResolver.Resolve(ctx, addr)
+	if err != nil {
+		return nil, err
+	}
+
+	if len(addrs) == 0 {
+		return nil, errors.New("non-resolvable API endpoint")
+	}
+
+	return addrs[0], nil
+}
--- a/core/commands/add.go
+++ b/core/commands/add.go
@ -8,6 +8,7 @@ import (
 	"strings"

 	core "github.com/ipfs/go-ipfs/core"
+	cmdenv "github.com/ipfs/go-ipfs/core/commands/cmdenv"
 	"github.com/ipfs/go-ipfs/core/coreunix"
 	filestore "github.com/ipfs/go-ipfs/filestore"
 	ft "gx/ipfs/QmQjEpRiwVvtowhq69dAtB4jhioPVFXiCcWZm9Sfgn7eqc/go-unixfs"
@ -141,7 +142,7 @@ You can now check what blocks have been created by:
 		return nil
 	},
 	Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) {
-		n, err := GetNode(env)
+		n, err := cmdenv.GetNode(env)
 		if err != nil {
 			res.SetError(err, cmdkit.ErrNormal)
 			return
--- a/core/commands/bitswap.go
+++ b/core/commands/bitswap.go
@ -7,6 +7,7 @@ import (

 	oldcmds "github.com/ipfs/go-ipfs/commands"
 	lgc "github.com/ipfs/go-ipfs/commands/legacy"
+	cmdenv "github.com/ipfs/go-ipfs/core/commands/cmdenv"
 	e "github.com/ipfs/go-ipfs/core/commands/e"
 	bitswap "gx/ipfs/QmTtmrK4iiM3MxWNA3pvbM9ekQiGZAiFyo57GP8B9FFgtz/go-bitswap"
 	decision "gx/ipfs/QmTtmrK4iiM3MxWNA3pvbM9ekQiGZAiFyo57GP8B9FFgtz/go-bitswap/decision"
@ -48,7 +49,7 @@ var unwantCmd = &oldcmds.Command{
 		}

 		if !nd.OnlineMode() {
-			res.SetError(errNotOnline, cmdkit.ErrClient)
+			res.SetError(ErrNotOnline, cmdkit.ErrClient)
 			return
 		}

@ -97,7 +98,7 @@ Print out all blocks currently on the bitswap wantlist for the local peer.`,
 		}

 		if !nd.OnlineMode() {
-			res.SetError(errNotOnline, cmdkit.ErrClient)
+			res.SetError(ErrNotOnline, cmdkit.ErrClient)
 			return
 		}

@ -140,14 +141,14 @@ var bitswapStatCmd = &cmds.Command{
 	},
 	Type: bitswap.Stat{},
 	Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) {
-		nd, err := GetNode(env)
+		nd, err := cmdenv.GetNode(env)
 		if err != nil {
 			res.SetError(err, cmdkit.ErrNormal)
 			return
 		}

 		if !nd.OnlineMode() {
-			res.SetError(errNotOnline, cmdkit.ErrClient)
+			res.SetError(ErrNotOnline, cmdkit.ErrClient)
 			return
 		}

@ -215,7 +216,7 @@ prints the ledger associated with a given peer.
 		}

 		if !nd.OnlineMode() {
-			res.SetError(errNotOnline, cmdkit.ErrClient)
+			res.SetError(ErrNotOnline, cmdkit.ErrClient)
 			return
 		}

@ -272,7 +273,7 @@ Trigger reprovider to announce our data to network.
 		}

 		if !nd.OnlineMode() {
-			res.SetError(errNotOnline, cmdkit.ErrClient)
+			res.SetError(ErrNotOnline, cmdkit.ErrClient)
 			return
 		}

--- a/core/commands/block.go
+++ b/core/commands/block.go
@ -10,6 +10,7 @@ import (
 	"os"

 	util "github.com/ipfs/go-ipfs/blocks/blockstoreutil"
+	cmdenv "github.com/ipfs/go-ipfs/core/commands/cmdenv"
 	e "github.com/ipfs/go-ipfs/core/commands/e"

 	"gx/ipfs/QmPTfgFTo9PFr1PvPKyKoeMgBvYPh6cX3aDP7DHKVbnCbi/go-ipfs-cmds"
@ -137,7 +138,7 @@ than 'sha2-256' or format to anything other than 'v0' will result in CIDv1.
 		cmdkit.IntOption("mhlen", "multihash hash length").WithDefault(-1),
 	},
 	Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) {
-		n, err := GetNode(env)
+		n, err := cmdenv.GetNode(env)
 		if err != nil {
 			res.SetError(err, cmdkit.ErrNormal)
 			return
@ -248,7 +249,7 @@ func getBlockForKey(ctx context.Context, env cmds.Environment, skey string) (blo
 		return nil, fmt.Errorf("zero length cid invalid")
 	}

-	n, err := GetNode(env)
+	n, err := cmdenv.GetNode(env)
 	if err != nil {
 		return nil, err
 	}
@ -282,7 +283,7 @@ It takes a list of base58 encoded multihashes to remove.
 		cmdkit.BoolOption("quiet", "q", "Write minimal output."),
 	},
 	Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) {
-		n, err := GetNode(env)
+		n, err := cmdenv.GetNode(env)
 		if err != nil {
 			res.SetError(err, cmdkit.ErrNormal)
 			return
--- a/core/commands/cat.go
+++ b/core/commands/cat.go
@ -7,6 +7,7 @@ import (
 	"os"

 	core "github.com/ipfs/go-ipfs/core"
+	cmdenv "github.com/ipfs/go-ipfs/core/commands/cmdenv"
 	coreunix "github.com/ipfs/go-ipfs/core/coreunix"

 	cmds "gx/ipfs/QmPTfgFTo9PFr1PvPKyKoeMgBvYPh6cX3aDP7DHKVbnCbi/go-ipfs-cmds"
@ -29,7 +30,7 @@ var CatCmd = &cmds.Command{
 		cmdkit.IntOption("length", "l", "Maximum number of bytes to read."),
 	},
 	Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) {
-		node, err := GetNode(env)
+		node, err := cmdenv.GetNode(env)
 		if err != nil {
 			res.SetError(err, cmdkit.ErrNormal)
 			return
--- a/core/commands/cmdenv/env.go
+++ b/core/commands/cmdenv/env.go
@ -1,4 +1,4 @@
-package commands
+package cmdenv

 import (
 	"fmt"
--- a/core/commands/dht.go
+++ b/core/commands/dht.go
@ -160,7 +160,7 @@ var findProvidersDhtCmd = &cmds.Command{
 		}

 		if n.Routing == nil {
-			res.SetError(errNotOnline, cmdkit.ErrNormal)
+			res.SetError(ErrNotOnline, cmdkit.ErrNormal)
 			return
 		}

@ -272,7 +272,7 @@ var provideRefDhtCmd = &cmds.Command{
 		}

 		if n.Routing == nil {
-			res.SetError(errNotOnline, cmdkit.ErrNormal)
+			res.SetError(ErrNotOnline, cmdkit.ErrNormal)
 			return
 		}

@ -424,7 +424,7 @@ var findPeerDhtCmd = &cmds.Command{
 		}

 		if n.Routing == nil {
-			res.SetError(errNotOnline, cmdkit.ErrNormal)
+			res.SetError(ErrNotOnline, cmdkit.ErrNormal)
 			return
 		}

@ -529,7 +529,7 @@ Different key types can specify other 'best' rules.
 		}

 		if n.Routing == nil {
-			res.SetError(errNotOnline, cmdkit.ErrNormal)
+			res.SetError(ErrNotOnline, cmdkit.ErrNormal)
 			return
 		}

@ -643,7 +643,7 @@ NOTE: A value may not exceed 2048 bytes.
 		}

 		if n.Routing == nil {
-			res.SetError(errNotOnline, cmdkit.ErrNormal)
+			res.SetError(ErrNotOnline, cmdkit.ErrNormal)
 			return
 		}

--- a/core/commands/dns.go
+++ b/core/commands/dns.go
@ -6,6 +6,7 @@ import (

 	cmds "github.com/ipfs/go-ipfs/commands"
 	e "github.com/ipfs/go-ipfs/core/commands/e"
+	ncmd "github.com/ipfs/go-ipfs/core/commands/name"
 	namesys "github.com/ipfs/go-ipfs/namesys"
 	nsopts "github.com/ipfs/go-ipfs/namesys/opts"

@ -72,7 +73,7 @@ The resolver can recursively resolve:
 			res.SetError(err, cmdkit.ErrNormal)
 			return
 		}
-		res.SetOutput(&ResolvedPath{output})
+		res.SetOutput(&ncmd.ResolvedPath{Path: output})
 	},
 	Marshalers: cmds.MarshalerMap{
 		cmds.Text: func(res cmds.Response) (io.Reader, error) {
@ -81,12 +82,12 @@ The resolver can recursively resolve:
 				return nil, err
 			}

-			output, ok := v.(*ResolvedPath)
+			output, ok := v.(*ncmd.ResolvedPath)
 			if !ok {
 				return nil, e.TypeErr(output, v)
 			}
 			return strings.NewReader(output.Path.String() + "\n"), nil
 		},
 	},
-	Type: ResolvedPath{},
+	Type: ncmd.ResolvedPath{},
 }
--- a/core/commands/files.go
+++ b/core/commands/files.go
@ -14,6 +14,7 @@ import (
 	oldcmds "github.com/ipfs/go-ipfs/commands"
 	lgc "github.com/ipfs/go-ipfs/commands/legacy"
 	core "github.com/ipfs/go-ipfs/core"
+	cmdenv "github.com/ipfs/go-ipfs/core/commands/cmdenv"
 	e "github.com/ipfs/go-ipfs/core/commands/e"
 	ft "gx/ipfs/QmQjEpRiwVvtowhq69dAtB4jhioPVFXiCcWZm9Sfgn7eqc/go-unixfs"
 	uio "gx/ipfs/QmQjEpRiwVvtowhq69dAtB4jhioPVFXiCcWZm9Sfgn7eqc/go-unixfs/io"
@ -114,7 +115,7 @@ var filesStatCmd = &cmds.Command{
 			res.SetError(err, cmdkit.ErrClient)
 		}

-		node, err := GetNode(env)
+		node, err := cmdenv.GetNode(env)
 		if err != nil {
 			res.SetError(err, cmdkit.ErrNormal)
 			return
@ -721,6 +722,7 @@ stat' on the file or any of its ancestors.
 	Options: []cmdkit.Option{
 		cmdkit.IntOption("offset", "o", "Byte offset to begin writing at."),
 		cmdkit.BoolOption("create", "e", "Create the file if it does not exist."),
+		cmdkit.BoolOption("parents", "p", "Make parent directories as needed."),
 		cmdkit.BoolOption("truncate", "t", "Truncate the file to size zero before writing."),
 		cmdkit.IntOption("count", "n", "Maximum number of bytes to read."),
 		cmdkit.BoolOption("raw-leaves", "Use raw blocks for newly created leaf nodes. (experimental)"),
@ -735,6 +737,7 @@ stat' on the file or any of its ancestors.
 		}

 		create, _ := req.Options["create"].(bool)
+		mkParents, _ := req.Options["parents"].(bool)
 		trunc, _ := req.Options["truncate"].(bool)
 		flush, _ := req.Options["flush"].(bool)
 		rawLeaves, rawLeavesDef := req.Options["raw-leaves"].(bool)
@ -745,7 +748,7 @@ stat' on the file or any of its ancestors.
 			return
 		}

-		nd, err := GetNode(env)
+		nd, err := cmdenv.GetNode(env)
 		if err != nil {
 			re.SetError(err, cmdkit.ErrNormal)
 			return
@ -757,6 +760,14 @@ stat' on the file or any of its ancestors.
 			return
 		}

+		if mkParents {
+			err := ensureContainingDirectoryExists(nd.FilesRoot, path, prefix)
+			if err != nil {
+				re.SetError(err, cmdkit.ErrNormal)
+				return
+			}
+		}
+
 		fi, err := getFileHandle(nd.FilesRoot, path, create, prefix)
 		if err != nil {
 			re.SetError(err, cmdkit.ErrNormal)
@ -1146,6 +1157,19 @@ func getPrefix(req oldcmds.Request) (cid.Builder, error) {
 	return &prefix, nil
 }

+func ensureContainingDirectoryExists(r *mfs.Root, path string, builder cid.Builder) error {
+	dirtomake := gopath.Dir(path)
+
+	if dirtomake == "/" {
+		return nil
+	}
+
+	return mfs.Mkdir(r, dirtomake, mfs.MkdirOpts{
+		Mkparents:  true,
+		CidBuilder: builder,
+	})
+}
+
 func getFileHandle(r *mfs.Root, path string, create bool, builder cid.Builder) (*mfs.File, error) {
 	target, err := mfs.Lookup(r, path)
 	switch err {
--- a/core/commands/filestore.go
+++ b/core/commands/filestore.go
@ -9,6 +9,7 @@ import (
 	oldCmds "github.com/ipfs/go-ipfs/commands"
 	lgc "github.com/ipfs/go-ipfs/commands/legacy"
 	"github.com/ipfs/go-ipfs/core"
+	cmdenv "github.com/ipfs/go-ipfs/core/commands/cmdenv"
 	e "github.com/ipfs/go-ipfs/core/commands/e"
 	"github.com/ipfs/go-ipfs/filestore"

@ -231,7 +232,7 @@ var dupsFileStore = &oldCmds.Command{
 }

 func getFilestore(env interface{}) (*core.IpfsNode, *filestore.Filestore, error) {
-	n, err := GetNode(env)
+	n, err := cmdenv.GetNode(env)
 	if err != nil {
 		return nil, nil, err
 	}
--- a/core/commands/get.go
+++ b/core/commands/get.go
@ -10,15 +10,16 @@ import (
 	"strings"

 	core "github.com/ipfs/go-ipfs/core"
+	cmdenv "github.com/ipfs/go-ipfs/core/commands/cmdenv"
 	e "github.com/ipfs/go-ipfs/core/commands/e"
-	uarchive "gx/ipfs/QmQjEpRiwVvtowhq69dAtB4jhioPVFXiCcWZm9Sfgn7eqc/go-unixfs/archive"
-	dag "gx/ipfs/QmRiQCJZ91B7VNmLvA6sxzDuBJGSojS3uXHHVuNr3iueNZ/go-merkledag"
-	path "gx/ipfs/QmdMPBephdLYNESkruDX2hcDTgFYhoCt4LimWhgnomSdV2/go-path"

 	"gx/ipfs/QmPTfgFTo9PFr1PvPKyKoeMgBvYPh6cX3aDP7DHKVbnCbi/go-ipfs-cmds"
 	"gx/ipfs/QmPtj12fdwuAqj9sBSTNUxBNu8kCGNp8b3o8yUzMm5GHpq/pb"
 	tar "gx/ipfs/QmQine7gvHncNevKtG9QXxf3nXcwSj6aDDmMm52mHofEEp/tar-utils"
+	uarchive "gx/ipfs/QmQjEpRiwVvtowhq69dAtB4jhioPVFXiCcWZm9Sfgn7eqc/go-unixfs/archive"
+	dag "gx/ipfs/QmRiQCJZ91B7VNmLvA6sxzDuBJGSojS3uXHHVuNr3iueNZ/go-merkledag"
 	"gx/ipfs/QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky/go-ipfs-cmdkit"
+	path "gx/ipfs/QmdMPBephdLYNESkruDX2hcDTgFYhoCt4LimWhgnomSdV2/go-path"
 )

 var ErrInvalidCompressionLevel = errors.New("compression level must be between 1 and 9")
@ -59,7 +60,7 @@ may also specify the level of compression by specifying '-l=<1-9>'.
 			return
 		}

-		node, err := GetNode(env)
+		node, err := cmdenv.GetNode(env)
 		if err != nil {
 			res.SetError(err, cmdkit.ErrNormal)
 			return
--- a/core/commands/keystore.go
+++ b/core/commands/keystore.go
@ -5,6 +5,7 @@ import (
 	"io"
 	"text/tabwriter"

+	cmdenv "github.com/ipfs/go-ipfs/core/commands/cmdenv"
 	"github.com/ipfs/go-ipfs/core/commands/e"
 	"github.com/ipfs/go-ipfs/core/coreapi/interface/options"

@ -66,7 +67,7 @@ var keyGenCmd = &cmds.Command{
 		cmdkit.StringArg("name", true, false, "name of key to create"),
 	},
 	Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) {
-		api, err := GetApi(env)
+		api, err := cmdenv.GetApi(env)
 		if err != nil {
 			res.SetError(err, cmdkit.ErrNormal)
 			return
@ -125,7 +126,7 @@ var keyListCmd = &cmds.Command{
 		cmdkit.BoolOption("l", "Show extra information about keys."),
 	},
 	Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) {
-		api, err := GetApi(env)
+		api, err := cmdenv.GetApi(env)
 		if err != nil {
 			res.SetError(err, cmdkit.ErrNormal)
 			return
@ -163,7 +164,7 @@ var keyRenameCmd = &cmds.Command{
 		cmdkit.BoolOption("force", "f", "Allow to overwrite an existing key."),
 	},
 	Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) {
-		api, err := GetApi(env)
+		api, err := cmdenv.GetApi(env)
 		if err != nil {
 			res.SetError(err, cmdkit.ErrNormal)
 			return
@ -215,7 +216,7 @@ var keyRmCmd = &cmds.Command{
 		cmdkit.BoolOption("l", "Show extra information about keys."),
 	},
 	Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) {
-		api, err := GetApi(env)
+		api, err := cmdenv.GetApi(env)
 		if err != nil {
 			res.SetError(err, cmdkit.ErrNormal)
 			return
--- a/core/commands/mount_unix.go
+++ b/core/commands/mount_unix.go
@ -91,7 +91,7 @@ baz

 		// error if we aren't running node in online mode
 		if node.LocalMode() {
-			res.SetError(errNotOnline, cmdkit.ErrClient)
+			res.SetError(ErrNotOnline, cmdkit.ErrClient)
 			return
 		}

--- a/core/commands/name/ipns.go
+++ b/core/commands/name/ipns.go
@ -1,20 +1,30 @@
-package commands
+package name

 import (
 	"errors"
+	"fmt"
 	"io"
 	"strings"
 	"time"

-	cmds "github.com/ipfs/go-ipfs/commands"
+	cmdenv "github.com/ipfs/go-ipfs/core/commands/cmdenv"
 	e "github.com/ipfs/go-ipfs/core/commands/e"
 	namesys "github.com/ipfs/go-ipfs/namesys"
 	nsopts "github.com/ipfs/go-ipfs/namesys/opts"

+	cmds "gx/ipfs/QmPTfgFTo9PFr1PvPKyKoeMgBvYPh6cX3aDP7DHKVbnCbi/go-ipfs-cmds"
+	logging "gx/ipfs/QmRREK2CAZ5Re2Bd9zZFG6FeYDppUWt5cMgsoUEp3ktgSr/go-log"
 	"gx/ipfs/QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky/go-ipfs-cmdkit"
 	offline "gx/ipfs/Qmd45r5jHr1PKMNQqifnbZy1ZQwHdtXUDJFamUEvUJE544/go-ipfs-routing/offline"
+	path "gx/ipfs/QmdMPBephdLYNESkruDX2hcDTgFYhoCt4LimWhgnomSdV2/go-path"
 )

+var log = logging.Logger("core/commands/ipns")
+
+type ResolvedPath struct {
+	Path path.Path
+}
+
 var IpnsCmd = &cmds.Command{
 	Helptext: cmdkit.HelpText{
 		Tagline: "Resolve IPNS names.",
@ -62,8 +72,8 @@ Resolve the value of a dnslink:
 		cmdkit.UintOption("dht-record-count", "dhtrc", "Number of records to request for DHT resolution."),
 		cmdkit.StringOption("dht-timeout", "dhtt", "Max time to collect values during DHT resolution eg \"30s\". Pass 0 for no timeout."),
 	},
-	Run: func(req cmds.Request, res cmds.Response) {
-		n, err := req.InvocContext().GetNode()
+	Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) {
+		n, err := cmdenv.GetNode(env)
 		if err != nil {
 			res.SetError(err, cmdkit.ErrNormal)
 			return
@ -77,8 +87,8 @@ Resolve the value of a dnslink:
 			}
 		}

-		nocache, _, _ := req.Option("nocache").Bool()
-		local, _, _ := req.Option("local").Bool()
+		nocache, _ := req.Options["nocache"].(bool)
+		local, _ := req.Options["local"].(bool)

 		// default to nodes namesys resolver
 		var resolver namesys.Resolver = n.Namesys
@ -98,7 +108,7 @@ Resolve the value of a dnslink:
 		}

 		var name string
-		if len(req.Arguments()) == 0 {
+		if len(req.Arguments) == 0 {
 			if n.Identity == "" {
 				res.SetError(errors.New("identity not loaded"), cmdkit.ErrNormal)
 				return
@ -106,12 +116,12 @@ Resolve the value of a dnslink:
 			name = n.Identity.Pretty()

 		} else {
-			name = req.Arguments()[0]
+			name = req.Arguments[0]
 		}

-		recursive, _, _ := req.Option("recursive").Bool()
-		rc, rcok, _ := req.Option("dht-record-count").Int()
-		dhtt, dhttok, _ := req.Option("dht-timeout").String()
+		recursive, _ := req.Options["recursive"].(bool)
+		rc, rcok := req.Options["dht-record-count"].(int)
+		dhtt, dhttok := req.Options["dht-timeout"].(string)
 		var ropts []nsopts.ResolveOpt
 		if !recursive {
 			ropts = append(ropts, nsopts.Depth(1))
@ -136,7 +146,7 @@ Resolve the value of a dnslink:
 			name = "/ipns/" + name
 		}

-		output, err := resolver.Resolve(req.Context(), name, ropts...)
+		output, err := resolver.Resolve(req.Context, name, ropts...)
 		if err != nil {
 			res.SetError(err, cmdkit.ErrNormal)
 			return
@ -144,21 +154,17 @@ Resolve the value of a dnslink:

 		// TODO: better errors (in the case of not finding the name, we get "failed to find any peer in table")

-		res.SetOutput(&ResolvedPath{output})
+		cmds.EmitOnce(res, &ResolvedPath{output})
 	},
-	Marshalers: cmds.MarshalerMap{
-		cmds.Text: func(res cmds.Response) (io.Reader, error) {
-			v, err := unwrapOutput(res.Output())
-			if err != nil {
-				return nil, err
-			}
-
+	Encoders: cmds.EncoderMap{
+		cmds.Text: cmds.MakeEncoder(func(req *cmds.Request, w io.Writer, v interface{}) error {
 			output, ok := v.(*ResolvedPath)
 			if !ok {
-				return nil, e.TypeErr(output, v)
+				return e.TypeErr(output, v)
 			}
-			return strings.NewReader(output.Path.String() + "\n"), nil
-		},
+			_, err := fmt.Fprintln(w, output.Path)
+			return err
+		}),
 	},
 	Type: ResolvedPath{},
 }
--- a/core/commands/name/ipnsps.go
+++ b/core/commands/name/ipnsps.go
@ -1,16 +1,18 @@
-package commands
+package name

 import (
 	"errors"
+	"fmt"
 	"io"
 	"strings"

-	cmds "github.com/ipfs/go-ipfs/commands"
-	e "github.com/ipfs/go-ipfs/core/commands/e"
+	"github.com/ipfs/go-ipfs/core/commands/cmdenv"
+	"github.com/ipfs/go-ipfs/core/commands/e"

-	peer "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer"
-	cmdkit "gx/ipfs/QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky/go-ipfs-cmdkit"
-	record "gx/ipfs/QmdHb9aBELnQKTVhvvA3hsQbRgUAwsWUzBP2vZ6Y5FBYvE/go-libp2p-record"
+	"gx/ipfs/QmPTfgFTo9PFr1PvPKyKoeMgBvYPh6cX3aDP7DHKVbnCbi/go-ipfs-cmds"
+	"gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer"
+	"gx/ipfs/QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky/go-ipfs-cmdkit"
+	"gx/ipfs/QmdHb9aBELnQKTVhvvA3hsQbRgUAwsWUzBP2vZ6Y5FBYvE/go-libp2p-record"
 )

 type ipnsPubsubState struct {
@ -21,6 +23,10 @@ type ipnsPubsubCancel struct {
 	Canceled bool
 }

+type stringList struct {
+	Strings []string
+}
+
 // IpnsPubsubCmd is the subcommand that allows us to manage the IPNS pubsub system
 var IpnsPubsubCmd = &cmds.Command{
 	Helptext: cmdkit.HelpText{
@ -42,26 +48,21 @@ var ipnspsStateCmd = &cmds.Command{
 	Helptext: cmdkit.HelpText{
 		Tagline: "Query the state of IPNS pubsub",
 	},
-	Run: func(req cmds.Request, res cmds.Response) {
-		n, err := req.InvocContext().GetNode()
+	Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) {
+		n, err := cmdenv.GetNode(env)
 		if err != nil {
 			res.SetError(err, cmdkit.ErrNormal)
 			return
 		}

-		res.SetOutput(&ipnsPubsubState{n.PSRouter != nil})
+		cmds.EmitOnce(res, &ipnsPubsubState{n.PSRouter != nil})
 	},
 	Type: ipnsPubsubState{},
-	Marshalers: cmds.MarshalerMap{
-		cmds.Text: func(res cmds.Response) (io.Reader, error) {
-			v, err := unwrapOutput(res.Output())
-			if err != nil {
-				return nil, err
-			}
-
+	Encoders: cmds.EncoderMap{
+		cmds.Text: cmds.MakeEncoder(func(req *cmds.Request, w io.Writer, v interface{}) error {
 			output, ok := v.(*ipnsPubsubState)
 			if !ok {
-				return nil, e.TypeErr(output, v)
+				return e.TypeErr(output, v)
 			}

 			var state string
@ -71,8 +72,9 @@ var ipnspsStateCmd = &cmds.Command{
 				state = "disabled"
 			}

-			return strings.NewReader(state + "\n"), nil
-		},
+			_, err := fmt.Fprintln(w, state)
+			return err
+		}),
 	},
 }

@ -80,8 +82,8 @@ var ipnspsSubsCmd = &cmds.Command{
 	Helptext: cmdkit.HelpText{
 		Tagline: "Show current name subscriptions",
 	},
-	Run: func(req cmds.Request, res cmds.Response) {
-		n, err := req.InvocContext().GetNode()
+	Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) {
+		n, err := cmdenv.GetNode(env)
 		if err != nil {
 			res.SetError(err, cmdkit.ErrNormal)
 			return
@ -106,11 +108,11 @@ var ipnspsSubsCmd = &cmds.Command{
 			paths = append(paths, "/ipns/"+peer.IDB58Encode(pid))
 		}

-		res.SetOutput(&stringList{paths})
+		cmds.EmitOnce(res, &stringList{paths})
 	},
 	Type: stringList{},
-	Marshalers: cmds.MarshalerMap{
-		cmds.Text: stringListMarshaler,
+	Encoders: cmds.EncoderMap{
+		cmds.Text: stringListMarshaler(),
 	},
 }

@ -118,8 +120,8 @@ var ipnspsCancelCmd = &cmds.Command{
 	Helptext: cmdkit.HelpText{
 		Tagline: "Cancel a name subscription",
 	},
-	Run: func(req cmds.Request, res cmds.Response) {
-		n, err := req.InvocContext().GetNode()
+	Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) {
+		n, err := cmdenv.GetNode(env)
 		if err != nil {
 			res.SetError(err, cmdkit.ErrNormal)
 			return
@ -130,7 +132,7 @@ var ipnspsCancelCmd = &cmds.Command{
 			return
 		}

-		name := req.Arguments()[0]
+		name := req.Arguments[0]
 		name = strings.TrimPrefix(name, "/ipns/")
 		pid, err := peer.IDB58Decode(name)
 		if err != nil {
@ -139,22 +141,17 @@ var ipnspsCancelCmd = &cmds.Command{
 		}

 		ok := n.PSRouter.Cancel("/ipns/" + string(pid))
-		res.SetOutput(&ipnsPubsubCancel{ok})
+		cmds.EmitOnce(res, &ipnsPubsubCancel{ok})
 	},
 	Arguments: []cmdkit.Argument{
 		cmdkit.StringArg("name", true, false, "Name to cancel the subscription for."),
 	},
 	Type: ipnsPubsubCancel{},
-	Marshalers: cmds.MarshalerMap{
-		cmds.Text: func(res cmds.Response) (io.Reader, error) {
-			v, err := unwrapOutput(res.Output())
-			if err != nil {
-				return nil, err
-			}
-
+	Encoders: cmds.EncoderMap{
+		cmds.Text: cmds.MakeEncoder(func(req *cmds.Request, w io.Writer, v interface{}) error {
 			output, ok := v.(*ipnsPubsubCancel)
 			if !ok {
-				return nil, e.TypeErr(output, v)
+				return e.TypeErr(output, v)
 			}

 			var state string
@ -164,7 +161,26 @@ var ipnspsCancelCmd = &cmds.Command{
 				state = "no subscription"
 			}

-			return strings.NewReader(state + "\n"), nil
-		},
+			_, err := fmt.Fprintln(w, state)
+			return err
+		}),
 	},
 }
+
+func stringListMarshaler() cmds.EncoderFunc {
+	return cmds.MakeEncoder(func(req *cmds.Request, w io.Writer, v interface{}) error {
+		list, ok := v.(*stringList)
+		if !ok {
+			return e.TypeErr(list, v)
+		}
+
+		for _, s := range list.Strings {
+			_, err := fmt.Fprintln(w, s)
+			if err != nil {
+				return err
+			}
+		}
+
+		return nil
+	})
+}
--- a/core/commands/name/name.go
+++ b/core/commands/name/name.go
@ -1,8 +1,7 @@
-package commands
+package name

 import (
-	cmds "github.com/ipfs/go-ipfs/commands"
-
+	"gx/ipfs/QmPTfgFTo9PFr1PvPKyKoeMgBvYPh6cX3aDP7DHKVbnCbi/go-ipfs-cmds"
 	"gx/ipfs/QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky/go-ipfs-cmdkit"
 )

--- a/core/commands/name/publish.go
+++ b/core/commands/name/publish.go
@ -1,26 +1,24 @@
-package commands
+package name

 import (
 	"context"
 	"errors"
 	"fmt"
 	"io"
-	"strings"
 	"time"

-	cmds "github.com/ipfs/go-ipfs/commands"
 	core "github.com/ipfs/go-ipfs/core"
+	cmdenv "github.com/ipfs/go-ipfs/core/commands/cmdenv"
 	e "github.com/ipfs/go-ipfs/core/commands/e"
 	keystore "github.com/ipfs/go-ipfs/keystore"
-	path "gx/ipfs/QmdMPBephdLYNESkruDX2hcDTgFYhoCt4LimWhgnomSdV2/go-path"

+	"gx/ipfs/QmPTfgFTo9PFr1PvPKyKoeMgBvYPh6cX3aDP7DHKVbnCbi/go-ipfs-cmds"
 	crypto "gx/ipfs/QmPvyPwuCgJ7pDmrKDxRtsScJgBaM5h4EpRL2qQJsmXf4n/go-libp2p-crypto"
 	peer "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer"
 	"gx/ipfs/QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky/go-ipfs-cmdkit"
+	path "gx/ipfs/QmdMPBephdLYNESkruDX2hcDTgFYhoCt4LimWhgnomSdV2/go-path"
 )

-var errNotOnline = errors.New("this command must be run in online mode. Try running 'ipfs daemon' first")
-
 var PublishCmd = &cmds.Command{
 	Helptext: cmdkit.HelpText{
 		Tagline: "Publish IPNS names.",
@ -73,8 +71,8 @@ Alternatively, publish an <ipfs-path> using a valid PeerID (as listed by
 		cmdkit.StringOption("ttl", "Time duration this record should be cached for (caution: experimental)."),
 		cmdkit.StringOption("key", "k", "Name of the key to be used or a valid PeerID, as listed by 'ipfs key list -l'. Default: <<default>>.").WithDefault("self"),
 	},
-	Run: func(req cmds.Request, res cmds.Response) {
-		n, err := req.InvocContext().GetNode()
+	Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) {
+		n, err := cmdenv.GetNode(env)
 		if err != nil {
 			res.SetError(err, cmdkit.ErrNormal)
 			return
@ -93,7 +91,7 @@ Alternatively, publish an <ipfs-path> using a valid PeerID (as listed by
 			return
 		}

-		pstr := req.Arguments()[0]
+		pstr := req.Arguments[0]

 		if n.Identity == "" {
 			res.SetError(errors.New("identity not loaded"), cmdkit.ErrNormal)
@ -102,9 +100,9 @@ Alternatively, publish an <ipfs-path> using a valid PeerID (as listed by

 		popts := new(publishOpts)

-		popts.verifyExists, _, _ = req.Option("resolve").Bool()
+		popts.verifyExists, _ = req.Options["resolve"].(bool)

-		validtime, _, _ := req.Option("lifetime").String()
+		validtime, _ := req.Options["lifetime"].(string)
 		d, err := time.ParseDuration(validtime)
 		if err != nil {
 			res.SetError(fmt.Errorf("error parsing lifetime option: %s", err), cmdkit.ErrNormal)
@ -113,8 +111,8 @@ Alternatively, publish an <ipfs-path> using a valid PeerID (as listed by

 		popts.pubValidTime = d

-		ctx := req.Context()
-		if ttl, found, _ := req.Option("ttl").String(); found {
+		ctx := req.Context
+		if ttl, found := req.Options["ttl"].(string); found {
 			d, err := time.ParseDuration(ttl)
 			if err != nil {
 				res.SetError(err, cmdkit.ErrNormal)
@ -124,7 +122,7 @@ Alternatively, publish an <ipfs-path> using a valid PeerID (as listed by
 			ctx = context.WithValue(ctx, "ipns-publish-ttl", d)
 		}

-		kname, _, _ := req.Option("key").String()
+		kname, _ := req.Options["key"].(string)
 		k, err := keylookup(n, kname)
 		if err != nil {
 			res.SetError(err, cmdkit.ErrNormal)
@ -142,22 +140,18 @@ Alternatively, publish an <ipfs-path> using a valid PeerID (as listed by
 			res.SetError(err, cmdkit.ErrNormal)
 			return
 		}
-		res.SetOutput(output)
+		cmds.EmitOnce(res, output)
 	},
-	Marshalers: cmds.MarshalerMap{
-		cmds.Text: func(res cmds.Response) (io.Reader, error) {
-			v, err := unwrapOutput(res.Output())
-			if err != nil {
-				return nil, err
-			}
+	Encoders: cmds.EncoderMap{
+		cmds.Text: cmds.MakeEncoder(func(req *cmds.Request, w io.Writer, v interface{}) error {
 			entry, ok := v.(*IpnsEntry)
 			if !ok {
-				return nil, e.TypeErr(entry, v)
+				return e.TypeErr(entry, v)
 			}

-			s := fmt.Sprintf("Published to %s: %s\n", entry.Name, entry.Value)
-			return strings.NewReader(s), nil
-		},
+			_, err := fmt.Fprintf(w, "Published to %s: %s\n", entry.Name, entry.Value)
+			return err
+		}),
 	},
 	Type: IpnsEntry{},
 }
--- a/core/commands/object/patch.go
+++ b/core/commands/object/patch.go
@ -7,6 +7,7 @@ import (

 	oldcmds "github.com/ipfs/go-ipfs/commands"
 	lgc "github.com/ipfs/go-ipfs/commands/legacy"
+	cmdenv "github.com/ipfs/go-ipfs/core/commands/cmdenv"
 	e "github.com/ipfs/go-ipfs/core/commands/e"
 	coreiface "github.com/ipfs/go-ipfs/core/coreapi/interface"
 	"github.com/ipfs/go-ipfs/core/coreapi/interface/options"
@ -67,7 +68,7 @@ the limit will not be respected by the network.
 		cmdkit.FileArg("data", true, false, "Data to append.").EnableStdin(),
 	},
 	Run: func(req *cmds.Request, re cmds.ResponseEmitter, env cmds.Environment) {
-		api, err := GetApi(env)
+		api, err := cmdenv.GetApi(env)
 		if err != nil {
 			re.SetError(err, cmdkit.ErrNormal)
 			return
@ -250,15 +251,3 @@ to a file containing 'bar', and returns the hash of the new object.
 		oldcmds.Text: objectMarshaler,
 	},
 }
-
-// TODO: fix import loop with core/commands so we don't need that
-// COPIED FROM ONE LEVEL UP
-// GetApi extracts CoreAPI instance from the environment.
-func GetApi(env cmds.Environment) (coreiface.CoreAPI, error) {
-	ctx, ok := env.(*oldcmds.Context)
-	if !ok {
-		return nil, fmt.Errorf("expected env to be of type %T, got %T", ctx, env)
-	}
-
-	return ctx.GetApi()
-}
--- a/core/commands/p2p.go
+++ b/core/commands/p2p.go
@ -405,7 +405,7 @@ func getNode(req cmds.Request) (*core.IpfsNode, error) {
 	}

 	if !n.OnlineMode() {
-		return nil, errNotOnline
+		return nil, ErrNotOnline
 	}

 	return n, nil
--- a/core/commands/ping.go
+++ b/core/commands/ping.go
@ -78,7 +78,7 @@ trip latency information.

 		// Must be online!
 		if !n.OnlineMode() {
-			res.SetError(errNotOnline, cmdkit.ErrClient)
+			res.SetError(ErrNotOnline, cmdkit.ErrClient)
 			return
 		}

--- a/core/commands/pubsub.go
+++ b/core/commands/pubsub.go
@ -11,6 +11,7 @@ import (
 	"time"

 	core "github.com/ipfs/go-ipfs/core"
+	cmdenv "github.com/ipfs/go-ipfs/core/commands/cmdenv"
 	e "github.com/ipfs/go-ipfs/core/commands/e"

 	cmds "gx/ipfs/QmPTfgFTo9PFr1PvPKyKoeMgBvYPh6cX3aDP7DHKVbnCbi/go-ipfs-cmds"
@ -73,7 +74,7 @@ This command outputs data in the following encodings:
 		cmdkit.BoolOption("discover", "try to discover other peers subscribed to the same topic"),
 	},
 	Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) {
-		n, err := GetNode(env)
+		n, err := cmdenv.GetNode(env)
 		if err != nil {
 			res.SetError(err, cmdkit.ErrNormal)
 			return
@ -81,7 +82,7 @@ This command outputs data in the following encodings:

 		// Must be online!
 		if !n.OnlineMode() {
-			res.SetError(errNotOnline, cmdkit.ErrClient)
+			res.SetError(ErrNotOnline, cmdkit.ErrClient)
 			return
 		}

@ -206,7 +207,7 @@ To use, the daemon must be run with '--enable-pubsub-experiment'.
 		cmdkit.StringArg("data", true, true, "Payload of message to publish.").EnableStdin(),
 	},
 	Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) {
-		n, err := GetNode(env)
+		n, err := cmdenv.GetNode(env)
 		if err != nil {
 			res.SetError(err, cmdkit.ErrNormal)
 			return
@ -214,7 +215,7 @@ To use, the daemon must be run with '--enable-pubsub-experiment'.

 		// Must be online!
 		if !n.OnlineMode() {
-			res.SetError(errNotOnline, cmdkit.ErrClient)
+			res.SetError(ErrNotOnline, cmdkit.ErrClient)
 			return
 		}

@ -253,7 +254,7 @@ To use, the daemon must be run with '--enable-pubsub-experiment'.
 `,
 	},
 	Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) {
-		n, err := GetNode(env)
+		n, err := cmdenv.GetNode(env)
 		if err != nil {
 			res.SetError(err, cmdkit.ErrNormal)
 			return
@ -261,7 +262,7 @@ To use, the daemon must be run with '--enable-pubsub-experiment'.

 		// Must be online!
 		if !n.OnlineMode() {
-			res.SetError(errNotOnline, cmdkit.ErrClient)
+			res.SetError(ErrNotOnline, cmdkit.ErrClient)
 			return
 		}

@ -310,7 +311,7 @@ To use, the daemon must be run with '--enable-pubsub-experiment'.
 		cmdkit.StringArg("topic", false, false, "topic to list connected peers of"),
 	},
 	Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) {
-		n, err := GetNode(env)
+		n, err := cmdenv.GetNode(env)
 		if err != nil {
 			res.SetError(err, cmdkit.ErrNormal)
 			return
@ -318,7 +319,7 @@ To use, the daemon must be run with '--enable-pubsub-experiment'.

 		// Must be online!
 		if !n.OnlineMode() {
-			res.SetError(errNotOnline, cmdkit.ErrClient)
+			res.SetError(ErrNotOnline, cmdkit.ErrClient)
 			return
 		}

--- a/core/commands/repo.go
+++ b/core/commands/repo.go
@ -11,6 +11,7 @@ import (

 	oldcmds "github.com/ipfs/go-ipfs/commands"
 	lgc "github.com/ipfs/go-ipfs/commands/legacy"
+	cmdenv "github.com/ipfs/go-ipfs/core/commands/cmdenv"
 	e "github.com/ipfs/go-ipfs/core/commands/e"
 	corerepo "github.com/ipfs/go-ipfs/core/corerepo"
 	fsrepo "github.com/ipfs/go-ipfs/repo/fsrepo"
@ -165,7 +166,7 @@ Version         string The repo version.
 		cmdkit.BoolOption("human", "Output sizes in MiB."),
 	},
 	Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) {
-		n, err := GetNode(env)
+		n, err := cmdenv.GetNode(env)
 		if err != nil {
 			res.SetError(err, cmdkit.ErrNormal)
 			return
--- a/core/commands/resolve.go
+++ b/core/commands/resolve.go
@ -9,6 +9,7 @@ import (
 	cmds "github.com/ipfs/go-ipfs/commands"
 	"github.com/ipfs/go-ipfs/core"
 	e "github.com/ipfs/go-ipfs/core/commands/e"
+	ncmd "github.com/ipfs/go-ipfs/core/commands/name"
 	ns "github.com/ipfs/go-ipfs/namesys"
 	nsopts "github.com/ipfs/go-ipfs/namesys/opts"
 	path "gx/ipfs/QmdMPBephdLYNESkruDX2hcDTgFYhoCt4LimWhgnomSdV2/go-path"
@ -16,10 +17,6 @@ import (
 	"gx/ipfs/QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky/go-ipfs-cmdkit"
 )

-type ResolvedPath struct {
-	Path path.Path
-}
-
 var ResolveCmd = &cmds.Command{
 	Helptext: cmdkit.HelpText{
 		Tagline: "Resolve the value of names to IPFS.",
@ -113,7 +110,7 @@ Resolve the value of an IPFS DAG path:
 				res.SetError(err, cmdkit.ErrNormal)
 				return
 			}
-			res.SetOutput(&ResolvedPath{p})
+			res.SetOutput(&ncmd.ResolvedPath{Path: p})
 			return
 		}

@ -132,7 +129,7 @@ Resolve the value of an IPFS DAG path:

 		c := node.Cid()

-		res.SetOutput(&ResolvedPath{path.FromCid(c)})
+		res.SetOutput(&ncmd.ResolvedPath{Path: path.FromCid(c)})
 	},
 	Marshalers: cmds.MarshalerMap{
 		cmds.Text: func(res cmds.Response) (io.Reader, error) {
@ -141,12 +138,12 @@ Resolve the value of an IPFS DAG path:
 				return nil, err
 			}

-			output, ok := v.(*ResolvedPath)
+			output, ok := v.(*ncmd.ResolvedPath)
 			if !ok {
 				return nil, e.TypeErr(output, v)
 			}
 			return strings.NewReader(output.Path.String() + "\n"), nil
 		},
 	},
-	Type: ResolvedPath{},
+	Type: ncmd.ResolvedPath{},
 }
--- a/core/commands/root.go
+++ b/core/commands/root.go
@ -1,6 +1,7 @@
 package commands

 import (
+	"errors"
 	"io"
 	"strings"

@ -8,6 +9,7 @@ import (
 	lgc "github.com/ipfs/go-ipfs/commands/legacy"
 	dag "github.com/ipfs/go-ipfs/core/commands/dag"
 	e "github.com/ipfs/go-ipfs/core/commands/e"
+	name "github.com/ipfs/go-ipfs/core/commands/name"
 	ocmd "github.com/ipfs/go-ipfs/core/commands/object"
 	unixfs "github.com/ipfs/go-ipfs/core/commands/unixfs"

@ -18,6 +20,8 @@ import (

 var log = logging.Logger("core/commands")

+var ErrNotOnline = errors.New("this command must be run in online mode. Try running 'ipfs daemon' first")
+
 const (
 	ApiOption = "api"
 )
@ -125,7 +129,7 @@ var rootSubcommands = map[string]*cmds.Command{
 	"log":       lgc.NewCommand(LogCmd),
 	"ls":        lgc.NewCommand(LsCmd),
 	"mount":     lgc.NewCommand(MountCmd),
-	"name":      lgc.NewCommand(NameCmd),
+	"name":      name.NameCmd,
 	"object":    ocmd.ObjectCmd,
 	"pin":       lgc.NewCommand(PinCmd),
 	"ping":      lgc.NewCommand(PingCmd),
@ -160,11 +164,11 @@ var rootROSubcommands = map[string]*cmds.Command{
 	"get": GetCmd,
 	"dns": lgc.NewCommand(DNSCmd),
 	"ls":  lgc.NewCommand(LsCmd),
-	"name": lgc.NewCommand(&oldcmds.Command{
-		Subcommands: map[string]*oldcmds.Command{
-			"resolve": IpnsCmd,
+	"name": &cmds.Command{
+		Subcommands: map[string]*cmds.Command{
+			"resolve": name.IpnsCmd,
 		},
-	}),
+	},
 	"object": lgc.NewCommand(&oldcmds.Command{
 		Subcommands: map[string]*oldcmds.Command{
 			"data":  ocmd.ObjectDataCmd,
--- a/core/commands/shutdown.go
+++ b/core/commands/shutdown.go
@ -3,6 +3,8 @@ package commands
 import (
 	"fmt"

+	cmdenv "github.com/ipfs/go-ipfs/core/commands/cmdenv"
+
 	cmds "gx/ipfs/QmPTfgFTo9PFr1PvPKyKoeMgBvYPh6cX3aDP7DHKVbnCbi/go-ipfs-cmds"
 	"gx/ipfs/QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky/go-ipfs-cmdkit"
 )
@ -12,7 +14,7 @@ var daemonShutdownCmd = &cmds.Command{
 		Tagline: "Shut down the ipfs daemon",
 	},
 	Run: func(req *cmds.Request, re cmds.ResponseEmitter, env cmds.Environment) {
-		nd, err := GetNode(env)
+		nd, err := cmdenv.GetNode(env)
 		if err != nil {
 			re.SetError(err, cmdkit.ErrNormal)
 			return
--- a/core/commands/stat.go
+++ b/core/commands/stat.go
@ -7,6 +7,8 @@ import (
 	"os"
 	"time"

+	cmdenv "github.com/ipfs/go-ipfs/core/commands/cmdenv"
+
 	humanize "gx/ipfs/QmPSBJL4momYnE7DcUyk2DVhD6rH488ZmHBGLbxNdhU44K/go-humanize"
 	cmds "gx/ipfs/QmPTfgFTo9PFr1PvPKyKoeMgBvYPh6cX3aDP7DHKVbnCbi/go-ipfs-cmds"
 	peer "gx/ipfs/QmQsErDt8Qgw1XrsXf2BpEzDgGWtB1YLsTAARBup5b6B9W/go-libp2p-peer"
@ -80,7 +82,7 @@ Example:
 	},

 	Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) {
-		nd, err := GetNode(env)
+		nd, err := cmdenv.GetNode(env)
 		if err != nil {
 			res.SetError(err, cmdkit.ErrNormal)
 			return
@ -88,7 +90,7 @@ Example:

 		// Must be online!
 		if !nd.OnlineMode() {
-			res.SetError(errNotOnline, cmdkit.ErrClient)
+			res.SetError(ErrNotOnline, cmdkit.ErrClient)
 			return
 		}

--- a/core/commands/swarm.go
+++ b/core/commands/swarm.go
@ -72,7 +72,7 @@ var swarmPeersCmd = &cmds.Command{
 		}

 		if n.PeerHost == nil {
-			res.SetError(errNotOnline, cmdkit.ErrClient)
+			res.SetError(ErrNotOnline, cmdkit.ErrClient)
 			return
 		}

@ -223,7 +223,7 @@ var swarmAddrsCmd = &cmds.Command{
 		}

 		if n.PeerHost == nil {
-			res.SetError(errNotOnline, cmdkit.ErrClient)
+			res.SetError(ErrNotOnline, cmdkit.ErrClient)
 			return
 		}

@ -291,7 +291,7 @@ var swarmAddrsLocalCmd = &cmds.Command{
 		}

 		if n.PeerHost == nil {
-			res.SetError(errNotOnline, cmdkit.ErrClient)
+			res.SetError(ErrNotOnline, cmdkit.ErrClient)
 			return
 		}

@ -331,7 +331,7 @@ var swarmAddrsListenCmd = &cmds.Command{
 		}

 		if n.PeerHost == nil {
-			res.SetError(errNotOnline, cmdkit.ErrClient)
+			res.SetError(ErrNotOnline, cmdkit.ErrClient)
 			return
 		}

@ -381,7 +381,7 @@ ipfs swarm connect /ip4/104.131.131.82/tcp/4001/ipfs/QmaCpDMGvV2BGHeYERUEnRQAwe3
 		addrs := req.Arguments()

 		if n.PeerHost == nil {
-			res.SetError(errNotOnline, cmdkit.ErrClient)
+			res.SetError(ErrNotOnline, cmdkit.ErrClient)
 			return
 		}

@ -446,7 +446,7 @@ it will reconnect.
 		addrs := req.Arguments()

 		if n.PeerHost == nil {
-			res.SetError(errNotOnline, cmdkit.ErrClient)
+			res.SetError(ErrNotOnline, cmdkit.ErrClient)
 			return
 		}

@ -593,7 +593,7 @@ Filters default to those specified under the "Swarm.AddrFilters" config key.
 		}

 		if n.PeerHost == nil {
-			res.SetError(errNotOnline, cmdkit.ErrNormal)
+			res.SetError(ErrNotOnline, cmdkit.ErrNormal)
 			return
 		}

@ -641,7 +641,7 @@ add your filters to the ipfs config file.
 		}

 		if n.PeerHost == nil {
-			res.SetError(errNotOnline, cmdkit.ErrNormal)
+			res.SetError(ErrNotOnline, cmdkit.ErrNormal)
 			return
 		}

@ -714,7 +714,7 @@ remove your filters from the ipfs config file.
 		}

 		if n.PeerHost == nil {
-			res.SetError(errNotOnline, cmdkit.ErrNormal)
+			res.SetError(ErrNotOnline, cmdkit.ErrNormal)
 			return
 		}

--- a/core/commands/urlstore.go
+++ b/core/commands/urlstore.go
@ -5,13 +5,14 @@ import (
 	"io"
 	"net/http"

+	cmdenv "github.com/ipfs/go-ipfs/core/commands/cmdenv"
 	filestore "github.com/ipfs/go-ipfs/filestore"
-	balanced "gx/ipfs/QmQjEpRiwVvtowhq69dAtB4jhioPVFXiCcWZm9Sfgn7eqc/go-unixfs/importer/balanced"
-	ihelper "gx/ipfs/QmQjEpRiwVvtowhq69dAtB4jhioPVFXiCcWZm9Sfgn7eqc/go-unixfs/importer/helpers"
-	trickle "gx/ipfs/QmQjEpRiwVvtowhq69dAtB4jhioPVFXiCcWZm9Sfgn7eqc/go-unixfs/importer/trickle"

 	cmds "gx/ipfs/QmPTfgFTo9PFr1PvPKyKoeMgBvYPh6cX3aDP7DHKVbnCbi/go-ipfs-cmds"
 	mh "gx/ipfs/QmPnFwZ2JXKnXgMw8CdBPxn7FWh6LLdjUjxV1fKHuJnkr8/go-multihash"
+	balanced "gx/ipfs/QmQjEpRiwVvtowhq69dAtB4jhioPVFXiCcWZm9Sfgn7eqc/go-unixfs/importer/balanced"
+	ihelper "gx/ipfs/QmQjEpRiwVvtowhq69dAtB4jhioPVFXiCcWZm9Sfgn7eqc/go-unixfs/importer/helpers"
+	trickle "gx/ipfs/QmQjEpRiwVvtowhq69dAtB4jhioPVFXiCcWZm9Sfgn7eqc/go-unixfs/importer/trickle"
 	cmdkit "gx/ipfs/QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky/go-ipfs-cmdkit"
 	chunk "gx/ipfs/QmXzBbJo2sLf3uwjNTeoWYiJV7CjAhkiA4twtLvwJSSNdK/go-ipfs-chunker"
 	cid "gx/ipfs/QmZFbDTY9jfSBms2MchvYM9oYRbAF19K7Pby47yDBfpPrb/go-cid"
@ -53,7 +54,7 @@ time.

 	Run: func(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) {
 		url := req.Arguments[0]
-		n, err := GetNode(env)
+		n, err := cmdenv.GetNode(env)
 		if err != nil {
 			res.SetError(err, cmdkit.ErrNormal)
 			return
--- a/dev.md
+++ b/dev.md
@ -1,62 +0,0 @@
-# go-ipfs development
-
-This is a simple description of where the codebase stands.
-
-There are multiple subpackages:
-
- `bitswap` - the block exchange
- `blocks` - handles dealing with individual blocks and sharding files
- `blockservice` - handles getting and storing blocks
- `cmd/ipfs` - cli ipfs tool - the main **entrypoint** atm
- `config` - load/edit configuration
- `core` - the core node, joins all the pieces
- `fuse/readonly` - mount `/ipfs` as a readonly fuse fs
- `importer` - import files into ipfs
- `merkledag` - merkle dag data structure
- `path` - path resolution over merkledag data structure
- `peer` - identity + addresses of local and remote peers
- `routing` - the routing system
- `routing/dht` - the DHT default routing system implementation
- `swarm` - connection multiplexing, many peers and many transports
- `util` - various utilities
-
-
-### What's done:
-
- merkle dag data structure
- path resolution over merkle dag
- local storage of blocks
- basic file import/export (`ipfs add`, `ipfs cat`)
- mounting `/ipfs` (try `{cat, ls} /ipfs/<path>`)
- multiplexing connections (tcp atm)
- peer addressing
- dht - impl basic kademlia routing
- bitswap - impl basic block exchange functionality
- crypto - building trust between peers in the network
- block splitting on import - Rabin fingerprints, etc
-
-### What's in progress:
-
- ipns - impl `/ipns` obj publishing + path resolution
- expose objects to the web at `http://ipfs.io/<path>`
-
-
-### What's next:
-
- version control - `commit` like data structure
- more...
-
-## Cool demos
-
-A list of cool demos to work towards
-
- boot a VM from an image in ipfs
- boot a VM from a filesystem tree in ipfs
- publish static websites directly from ipfs
- expose objects to the web at `http://ipfs.io/<path>`
- mounted auto-committing versioned personal dropbox
- mounted encrypted personal/group dropbox
- mounted {npm, apt, other pkg manager} registry
- open a video on ipfs, stream it in
- watch a video with a topology of 1 seed N leechers (N ~100)
- more in section 3.8 in the [paper](https://github.com/ipfs/ipfs/blob/master/papers/ipfs-cap2pfs/ipfs-p2p-file-system.pdf)
--- a/docs/cli-http-api-core-diagram.png
+++ b/docs/cli-http-api-core-diagram.png
--- a/docs/gateway.md
+++ b/docs/gateway.md
@ -0,0 +1,48 @@
+# Gateway
+
+An IPFS Gateway acts as a bridge between traditional web browsers and IPFS.
+Through the gateway, users can browse files and websites stored in IPFS as if
+they were stored in a traditional web server.
+
+By default, go-ipfs nodes run a gateway at `http://127.0.0.1:5001/`.
+
+We also provide a public gateway at `https://ipfs.io`. If you've ever seen a
+link in the form `https://ipfs.io/ipfs/Qm...`, that's being served from *our*
+gateway.
+
+## Configuration
+
+The gateway's configuration options are (briefly) described in the
+[config](https://github.com/ipfs/go-ipfs/blob/master/docs/config.md#gateway)
+documentation.
+
+## Directories
+
+For convenience, the gateway (mostly) acts like a normal web-server when serving
+a directory:
+
+1. If the directory contains an `index.html` file:
+  1. If the path does not end in a `/`, append a `/` and redirect. This helps
+     avoid serving duplicate content from different paths.<sup>&dagger;</sup>
+  2. Otherwise, serve the `index.html` file.
+2. Dynamically build and serve a listing of the contents of the directory.
+
+<sub><sup>&dagger;</sup>This redirect is skipped if the query string contains a
+`go-get=1` parameter. See [PR#3964](https://github.com/ipfs/go-ipfs/pull/3963)
+for details</sub>
+
+## Filenames
+
+When downloading files, browsers will usually guess a file's filename by looking
+at the last component of the path. Unfortunately, when linking *directly* to a
+file (with no containing directory), the final component is just a CID
+(`Qm...`). This isn't exactly user-friendly.
+
+To work around this issue, you can add a `filename=some_filename` parameter to
+your query string to explicitly specify the filename. For example:
+
+> https://ipfs.io/ipfs/QmfM2r8seH2GiRaC4esTjeraXEachRt8ZsSeGaWTPLyMoG?filename=hello_world.txt
+
+## MIME-Types
+
+TODO
--- a/package.json
+++ b/package.json
@ -539,6 +539,12 @@
      "hash": "QmPyxJ2QS7L5FhGkNYkNcXHGjDhvGHueJ4auqAstFHYxy5",
      "name": "go-cidutil",
      "version": "0.0.2"
+    },
+    {
+      "author": "lgierth",
+      "hash": "QmfXU2MhWoegxHoeMd3A2ytL2P6CY4FfqGWc23LTNWBwZt",
+      "name": "go-multiaddr-dns",
+      "version": "0.2.4"
    }
  ],
  "gxVersion": "0.10.0",
--- a/test/sharness/t0236-cli-api-dns-resolve.sh
+++ b/test/sharness/t0236-cli-api-dns-resolve.sh
@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2015 Jeromy Johnson
+# MIT Licensed; see the LICENSE file in this repository.
+#
+
+test_description="test dns resolution of api endpoint by cli"
+
+. lib/test-lib.sh
+
+test_init_ipfs
+
+test_expect_success "can make http request against dns resolved nc server" '
+  nc -ld 5005 > nc_out &
+  NCPID=$!
+  go-sleep 0.5s && kill "$NCPID" &
+  ipfs cat /ipfs/Qmabcdef --api /dns4/localhost/tcp/5005 || true
+'
+
+test_expect_success "request was received by local nc server" '
+  grep "POST /api/v0/cat" nc_out
+'
+
+test_done
--- a/test/sharness/t0250-files-api.sh
+++ b/test/sharness/t0250-files-api.sh
@ -597,9 +597,29 @@ test_files_api() {
    ipfs files ls /adir | grep foobar
  '

+  test_expect_success "should fail to write file and create intermediate directories with no --parents flag set $EXTRA" '
+    echo "ipfs rocks" | test_must_fail ipfs files write --create /parents/foo/ipfs.txt
+  '
+
+  test_expect_success "can write file and create intermediate directories $EXTRA" '
+    echo "ipfs rocks" | ipfs files write --create --parents /parents/foo/bar/baz/ipfs.txt &&
+    ipfs files stat "/parents/foo/bar/baz/ipfs.txt" | grep -q "^Type: file"
+  '
+
+  test_expect_success "can write file and create intermediate directories with short flags $EXTRA" '
+    echo "ipfs rocks" | ipfs files write -e -p /parents/foo/bar/baz/qux/quux/garply/ipfs.txt &&
+    ipfs files stat "/parents/foo/bar/baz/qux/quux/garply/ipfs.txt" | grep -q "^Type: file"
+  '
+
+  test_expect_success "can write another file in the same directory with -e -p $EXTRA" '
+    echo "ipfs rocks" | ipfs files write -e -p /parents/foo/bar/baz/qux/quux/garply/ipfs2.txt &&
+    ipfs files stat "/parents/foo/bar/baz/qux/quux/garply/ipfs2.txt" | grep -q "^Type: file"
+  '
+
  test_expect_success "clean up $EXTRA" '
    ipfs files rm -r /foobar &&
-    ipfs files rm -r /adir
+    ipfs files rm -r /adir &&
+    ipfs files rm -r /parents
  '

  test_expect_success "root mfs entry is empty $EXTRA" '
--- a/test/sharness/t0272-urlstore.sh
+++ b/test/sharness/t0272-urlstore.sh
@ -57,6 +57,7 @@ test_expect_success "make sure hashes are different" '
 '

 test_expect_success "get files via urlstore" '
+  rm -f file1.actual file2.actual &&
  ipfs get $HASH1 -o file1.actual &&
  test_cmp file1 file1.actual &&
  ipfs get $HASH2 -o file2.actual &&
@ -111,6 +112,11 @@ test_expect_success "files can not be retrieved via the urlstore" '
  test_must_fail ipfs cat $HASH2 > /dev/null
 '

+test_expect_success "remove broken files" '
+  ipfs pin rm $HASH1 $HASH2 &&
+  ipfs repo gc > /dev/null
+'
+
 test_expect_success "add large file using gateway address via url store" '
  HASH3=$(ipfs urlstore add http://127.0.0.1:$GWAY_PORT/ipfs/$HASH3a)
 '
@ -120,6 +126,7 @@ test_expect_success "make sure hashes are different" '
 '

 test_expect_success "get large file via urlstore" '
+  rm -f file3.actual &&
  ipfs get $HASH3 -o file3.actual &&
  test_cmp file3 file3.actual
 '