Channelized I/O, intent-based security policies (e.g. from RACF), multi-layered error handling and recovery. It's a rich garden. However, it's not as simple as "implementing a feature," since the interesting things are all systems of activity with interlocking assumptions and expectations with all the other systems. You can't "just" pluck pieces out of context any more than you can grab a cool phrase from Mandarin or Urdu, or admire a lobster's claw and decide to graft it onto your own arm.
But for anyone interested in evolving systems/OSs, definitely study S/3x0 and Z successors, or the proprietary mainframes and minicomputers in general. In many cases we are now stumbling into reinventing techniques that mainframes or minicomputer teams built many years earlier. Best case in point probably virtual machines (VMs), in which VMware et al started in ~2003 rebuilding a technology capability that had been developed in Z systems in 1967/68.
It's essentially having a small CPU as a DMA controller. For a prototypical DMA controller, you just give a list of source and destination addresses + transfer size and it runs off an performs those transfers and tells you when it's done. Maybe they can even be chained together (one DMA channel writes to control registers of another then starts that channel).
For "channelized" IO, you had CPU instructions which would effectively hand off small programs to another processor that was extremely limited in capability compared to your CPU. The channel processor would handle the direct interrupts/events from physical IO devices, do basic processing, then kick off (a) DMA transfer(s) to and from main memory, or as a data stream straight to the CPU.
For some mainframe architectures, you could implement things like text-editors and filesystem drivers that would run on the channel processors so that basic tasks didn't take up core CPU time. The main CPU could allocate memory for a process to be placed in, then send off a channel program to the tape drive and go and do something else for awhile while the tape drive found and loaded the executable completely independent of the CPU.
Probably a more realistic example would be to take something like a database and have a separate CPU processing the on-disk format, or a separate CPU to process your network protocol's wire format and only having the actual data it contains seen by the main CPU.
These days processing power is so ridiculously cheap compared to those days that flexibility rules the day. Might as well have a bunch of dumb IO devices because even a basic CPU core can move and process gigabytes of data per second.
The channel I/O format has conditional jumps in the command stream and lets you as the application programmer offload large parts of the filesystem or database directly onto the hard drive controller without main CPU intervention. It's closer to GPU command submissions than what you see out of NVMe/SCSI/ATA. So, for one example, telling the controller to "walk an index tree in this format and retrieve the block that matches this key" is something you can code yourself in the channel I/O command stream as it's own sort of ISA.
From the "roads not taken" perspective, Intel actually made something like a channel IO engine for the 8086/8088 family called the 8089 (https://en.wikipedia.org/wiki/Intel_8089) which supported the 20-bit address space of those systems and proper 8 and 16-bit operations. You could write little programs for it which would communicate with IO devices and respond to interrupts on either a shared or private bus, do some processing, and then move the results to main memory.
IBM in a cost saving move grafted the DMA engine from the 8080 family (the 8237) and made it mostly work by adding a page register to cover the remaining bits and setting one up in an odd way for the 286 to get it to do IO <-> memory transfers for 16-bit devices (but unable to do memory-to-memory transfers).
See eg https://netfpga.org for pretty much exactly this for network. I believe it's more of a research platform than something that's used in production. It's not exactly new either.
The new "Z on demand" features they offer over the internet remind me a lot of how it easy it was to provision virtual S/360 (S/390 at the time) hardware and software with VM/CMS when I worked there in the 90's.
I've been playing with VM/370 and MVS 3,8j on Hercules for some time, but I haven't figured that out yet. The concepts are so alien for a Unix person I may have gone over the instructions a couple times, without recognizing them as such.
Agree on ISPF being a great editor. It was trivial for new users to pick up, but at the same time had a rich feature set.
My favorite part was its support for folds, or what it called "excluded" lines. You could issue an initial command that excluded lines you wanted to ignore, and then issue subsequent commands to operate on lines not excluded, or "NX". Very nice. I occasionally wish I had ISPF while I'm in the middle of a Vim session.
The most fun thing in VM/CMS is REXX. It's an awesome language invented by Mike Colishaw with the express goal of being easy to program above all other considerations. Its PARSE instruction is one of the most powerful things I've ever seen in any language.
REXX was so great it became the standard scripting language for all IBM system products and was incorporated into the OS command line and the text editor, XEDIT. This meant you could have one language that ran commands and programs in any other language, could do anything at the command line (like create machines, etc), and could edit and save text files. Think about that for a second. It was WAAAAY ahead of its time.
Sadly, REXX predated the internet and never had a browser-savvy release. A miscarriage of REXX and OOPS called Object Rexx was also unsuccessful.
Presumably, most anything worthwhile would still be in Z/OS from whence it would have pretty naturally made its way into IBM's Unix and eventually Linux.
OS/360 was quite different from most modern operating systems--most notably it was batch and designed for very small memory sizes--but different isn't really better in this case.
I've looked at it only briefly, but... From my (mostly Unix-ish) perspective z/OS looks a bit like as if MS-DOS have been continuously developed by a few hundred people since the eighties. There are some nice things, but the overall system architecture is somewhat... nonexistant.
But for anyone interested in evolving systems/OSs, definitely study S/3x0 and Z successors, or the proprietary mainframes and minicomputers in general. In many cases we are now stumbling into reinventing techniques that mainframes or minicomputer teams built many years earlier. Best case in point probably virtual machines (VMs), in which VMware et al started in ~2003 rebuilding a technology capability that had been developed in Z systems in 1967/68.