Staking Networks Are Building Networks Like Apps and Without Devops)

A proof of stake network is a network. It's not an application running on a network. That seems obvious, doesn't it?

Yet my experience so far is that staking networks are building networks like they'd build an app. They're not building them like networks.

As a staking network validator with experience designing and operating mission critical networks, I've been thinking about this divergence in terms of operational requirements.

The staking networks coming online have very high valuations. This means they should be held to the highest standards of availability, stability and security. Building a network like an app can't meet the standards these networks need to adhere to.

Top Down Without the Bottom Up

Admittedly, the introduction of software defined networking has collapsed the traditional telecommunications networking stack. This is true whether you believe the OSI or TCP/IP stack is the "true" network stack.

When I think of a network, I start at the bottom. I start at the physical cable in the ground. Applications sit at the top of the stack. The physical connectivity gets abstracted away the further up the stack you go.

Developing apps starts at the top of the stack. My sense is most times app developers don't think too much about the lowest networking layers. They're intended to be invisible anyway.

Staking networks bridge this gap to some extent. I'm hearing more and more people refer to a staking network's layer 1 as the "protocol" layer. This strikes me as abstract, as I was trained to think of layer 1 as the actual physical connectivity layer. Plus, "protocol" means different things to different people.

As a result, software developers start at the top and work their way down. My sense is that most staking project developers are working in lower levels of the stack than they're accustomed to.

What's missing is the complementary work, starting at the bottom and working upwards on the stack. This is where devops usually does their thing. I haven't come across many, if any, devops people working at staking projects.

This realization got me thinking about this paradox. Staking networks are building networks with software developers. This results in networks being built like apps.

Closing the Gap

Validators have evolved by default into filling this gap. This can work to some extent, especially if the set is populated with validators who have worked in mission critical networking environments. As in many roles, the experience gained in working in these environments can't be gained any other way.

It's not enough though.

The current approach means these networks aren't paying the attention necessary to the lower levels of the communication stack. Just like software development is an art to developers, lower level network architectures are the devops practitioner's art.

(As a diversion, if you want to get a feel for how hardcore the "old school" telecom operators were, take a look at this AT&T spec for tying down a cable. The cable tie down had to be a certain type of string tied with a certain knot. It COULDN'T be a nylon zip tie. See section 2.9 here for details.)

(I once made the mistake of trying to use a nylon zip tie in the NYC data center room of a telco. The room was managed by a long time phone company employee. He almost kicked me out and banned me from the room for doing that.

It turns out that over the time frame the telecom networks were expected to last, a nylon zip tie would end up slicing through a cable's protective covering and ultimately the cable itself.)

Falling Short of Mission Critical

Take the recent aborted Cosmos upgrade attempt as an example.

No mission critical network I've ever worked on would have tried to upgrade without having thoroughly tested the upgrade on a test network and having a documented roll-back plan in place. For example, financial institutions  know exactly how much money they lose per second of network downtime. An hour of downtime, like the cosmos-hub experienced, would be completely unacceptable in the legacy financial world.

Admittedly, staking networks are in their infancy and we're all learning together as we go. I'm writing this to bring awareness to a weakness I'm seeing.

DevOps are the Missing Piece

In order for staking networks to be seen as viable alternatives to the legacy financial system, networks need to complement their top down approach to building with a bottom up one.

The first step is to hire one or two devops resources, with mission critical operational experience. The perspective they bring will go a long way toward increasing the security, stability and availability of staking networks, both now and in the long run. My sense is doing so would also have the welcome side effect of strengthening communication between staking projects and validators in the process.

P.S. - Need help bridging the gap between your network and validators? I help networks do that. Contact me here to start the discussion.