“The Third University of Cambridge”: BBN and the Development of the ARPAnet
We’ve all heard that “DARPA invented the Internet.” But few have heard of BBN, the contractor that did the most work to bring the ARPAnet into existence. Today’s piece dives into the history of BBN and the firm’s unique structure. A firm like BBN winning the main portion of the ARPAnet project was a pivotal reason the ARPAnet project went so smoothly. BBN embodied the “middle ground between academia and the commercial world.” BBN’s early operating model provides an ideal management framework for anyone looking to deploy researchers on ambitious research projects within the structure of a firm. With BBN’s structure, many difficult projects become possible.
With that, let’s get into it.
Introduction
DARPA has no lab benches, research facilities, or staff scientists doing science. Instead, it has contractors. DARPA uses contractor ecosystem to execute any ideas the office chooses to fund. Throughout this series I will generally write project histories from the POV of DARPA’s PMs and office directors. This is a natural choice because, depending on the era of the organization, either the project managers or the office directors are usually tasked with driving projects forward.
However, in the coming trilogy of pieces I’ll tell stories from the POV of some of DARPA’s best contractors ever. Projects are selected, funded, and often steered by the DARPA office; but ideas are entirely executed by contractors.
In many cases, which contractor wins a contract might only have a moderate impact on the cost efficiency, speed, and overall results of a project. In those cases, it might be hard for a contractor — whether it be a university research lab, private company, or a non-profit institute like RAND — to prove it is truly top-notch.
One the other hand, there are cases where many in a field do not even bid on a contract because they do not find the project specs feasible. In these cases, a level of excellence can be required to deliver on a proposal. This is the category the now-famous ARPAnet fell into. As most readers know, the ARPAnet was the precursor to the internet. It was as the main contractor on this project that the MIT faculty spin-off research firm Bolt, Beranek, & Newman (BBN) would cement its name in the history books.
The coming trilogy of pieces exploring DARPA contracts from the contractors’ POV provide an essential perspective to this series. Contractors’ work on projects like BBN with the ARPAnet, CMU’s driverless vehicle projects, and Lockheed Skunk Work’s STEALTH project have many management insights in common — despite the obvious differences between those projects and contractors.
Today’s piece dives into the contractor primarily responsible for DARPA’s most famous project: BBN.
What was BBN?
BBN’s stellar results on the ARPAnet project did not shock many in the computing community. In the late 1960s, the firm was known to be special. It was an interesting kind of research firm, of a sort we don’t see much of today. BBN’s staff found themselves natural members of the elite Cambridge, Massachusetts research community despite the firm’s for-profit status. The firm was home to some of the best theoretical researchers in Cambridge — formerly professors at Harvard and MIT — as well as top-tier engineering researchers who often defected from the large computing projects at MIT’s Lincoln Lab.
In general, many researchers defected because they believed BBN was a better place to work on big problems in more exciting ways than at Lincoln. Theoretically inclined researchers, like Robert Kahn, found this a great place to do proper research in an applied context. Great engineers who cared less for theory, like the project’s lead engineer Frank Heart, found the firm an ideal structure to work on implementing real technology that still had an extreme level of novelty.
To be sure, BBN was a company that found ways to make a profit and grow at a considerable rate in its early decades — 26% year over year. However, it was also a firm that went out of its way to ensure that its researchers were happy. It wanted the best problems — as long as it could cover the costs and salaries of the project with some kind of funding — not necessarily the most profitable problems.
Robert Kahn and J.C.R. Licklider are two familiar names that called BBN home for a time. From his view as IPTO chief and minor celebrity in the computing research community in his day, Robert Kahn had a deep understanding of what went on at every notable computer research center in the world. So it should be taken as no small thing when, in an oral history, Kahn had this extreme praise for the firm in this era:
BBN was a kind of hybrid version of Harvard and MIT in the sense that most of the people there were either faculty or former faculty at either Harvard or MIT. If you've ever spent any time at either of those places, you would know what a unique kind of organization BBN was. A lot of the students at those places spent time at BBN. It was kind of like a super hyped-up version of the union of the two, except that you didn't have to worry about classes and teaching. You could just focus on research. It was sort of the cognac of the research business, very distilled. The culture at BBN at the time was to do interesting things and move on to the next interesting thing. There was more incentive to come up with interesting ideas and explore them than to try to capitalize on them once they had been developed.
As we’ll explore, some at the firm would often try to capitalize on big ideas. But this was not the focus of the firm.
From a legal point of view, BBN was legally structured as a partnership for its early decades — kind of like a law firm — before going public in 1966. Teams at BBN were not exactly permanent. Individuals were often recruited simply because BBNers felt they could contribute to some coming project, or simply “raise the average level of competence of the firm.”
The firm’s culture was far more similar to a university lab than an incumbent firm like Honeywell. If a talented researcher found an area of exploration interesting, the firm did what it could to find a payer for that research. For example, this was the case with a young J.C.R. Licklider looking to further explore the world of interactive personal computing on a very expensive machine. Bolt and Beranek did not say no. Rather, they said yes, bought a machine, and found a way to fund the project through a grant from the Ford Foundation. The long research report — later turned into a book — by Licklider and other BBNers titled, “Libraries of the Future” was far more than it seems. The report did more than just carry out a forward-looking analysis by information researchers and engineers assessing what digitized libraries could look like for a library non-profit. The report contained much of the early technical exploration that would grow to become Licklider’s legacy.
Beyond BBN’s whole-hearted embrace of the “research” in “research firm,” its structure as a “firm” allowed it to carry out extended, elaborate implementation projects like the ARPAnet project that were ill-suited to an academic lab. The firm’s expertise in early computing systems — built up by contract research projects as well as the team’s pre-existing experience at places like Lincoln Lab — allowed it to push technology areas forward in a self-funded way. For example, in the 1960s the firm designed, implemented, and maintained one of the first commercial time-sharing systems for non-engineers. The project gave Massachusetts General Hospital a time-sharing system. The system allowed the hospital to handle patient administration in a modern way and automate some of the data analysis in their research. This, of course, was not the first time-sharing system. But projects like this allowed BBN engineers to continue honing expertise with young technology and earn private sector wages in doing so.
The ARPAnet project was another implementation project optimally suited to BBN’s somewhat unique structure and makeup. But, to gain a full understanding of how the firm came to be what it was, let’s take a step back and start at the beginning. BBN was not started as a computing research firm. Bolt and Beranek were not computer scientists. They were acoustics professors who, steeped in the applied research tradition of early MIT, began to become overwhelmed by their contract research work in the 1940s.
BBN’s early years as an acoustics firm
BBN began in a fashion not unlike many early-20th-century MIT spin-off companies. A team of professors and graduate students worked extensively on consulting and contract work. As time went on and word spread, the demand for their contracting services grew. For a while, renting an additional room in their lab building and hiring more graduate students was enough to keep up with demand. Eventually, the young company had to get its own quarters and hire its own full-time employees.
In the case of BBN, the original professors involved were involved in the relatively young area of psycho-acoustics. In the wake of World War II, Richard Bolt was set to become the director of a new acoustics laboratory at the Institute to continue several courses of research that were largely funded by the Navy. Faculty and staff with backgrounds in physics, electrical engineering, architecture, mechanical engineering, aeronautical engineering, and psychology would all contribute to the contracts. In these post-war years, Leo Beranek was recruited to join the lab as its Technical Director from Harvard’s electro-acoustic laboratory. He was also given a professorship.
Since the inception of MIT’s Technology Plan in 1920, the Institute had made a habit of actively facilitating research contracts to its faculty. Professor’s contract salaries covered only four days a week during the academic year and left weekends, holidays, and summers entirely free to the professors for contract work. Professors could wholeheartedly spend their time on contract work with close to zero red tape or permission required from MIT. Unlike places like Harvard — and most universities — the Institute, in this era, encouraged this double life for its professors. Negligible amounts of paperwork and permission-asking were involved. MIT saw no more suitable activity for faculty at a professionally-focused institute of technology than to ply their craft for paying customers. In doing so, professors simultaneously kept up with industry trends, solved industry’s problems, learned how to keep their university research industry-relevant, and brought money into the Institute.
In the heyday of this arrangement, requests from industry regularly came into MIT’s office of the president for help in various problems, such as acoustics. When acoustics queries came in, they were routed to Richard Bolt. In 1946, one of these requests came from an architecture firm responsible for building the UN Headquarters in New York City. Bolt submitted a bid and won the commission. In 1948, the drawings arrived from the firm with the request for Bolt to begin his acoustics consulting work. Given the massive size of the drawings, Bolt realized the project was not a one-man job. He asked Leo Beranek if he’d like to work on the contract as a team. The two had papers drawn up, and the partnership of Bolt & Beranek came into existence.
For this first contract, Bolt and Robert Newman — one of four part-time graduate students hired to work on the project, who was brought into the partnership upon graduation — were responsible for figuring out acoustical treatments for the UN building. Beranek was in charge of the more troublesome task of sound system design and equipment selection for the unorthodox building. Despite the difficulty, the project turned out to be a very public success. Beranek wrote that the firm’s “name was known to architects everywhere, and business boomed.”
To this point, the firm had simply been run out of an extra room in the Acoustics Laboratory. As Beranek described the arrangement:
The firm, Bolt & Beranek, had the blessings of MIT’s new President, James Killian. He offered to help us get started and rented us two rooms in the MIT Acoustics Laboratory for our use, but warned us that we would have to seek space outside of MIT if our needs expanded.
By the end of the business’ first twelve months, the room was already full of equipment the firm had bought to continue to service the growing number of contracts.
In late 1949, the young firm moved to another address in Cambridge. Gradually growing, in 1951 the firm would move again to take up two apartments and a basement in a mostly residential building on Elliot Street. By 1956, the acoustics-focused firm had built up to 50 full-time employees — many of them former Cambridge graduate students — along with an additional office in the architecture hotbed that was LA. This all happened without outside financing beyond the firm’s line of credit with the local bank.
In this period, the firm also added two more of the four founding part-time graduate students to its partnership. These new partners, Jordan Baruch and Samuel Labate, had worked with the firm for years and also completed graduate theses in areas relevant to the firm. Additionally, in 1953, BBN incorporated in order to shield itself from liabilities associated with the growing subsection of its business associated with controlling aircraft noise. The National Advisory Committee on Aeoronautics and companies that manufactured jet engines hired the firm to work on problems such as designing structures to minimize noise during engine testing. Projects like this had many practical uses, but also represented cutting-edge uses of the theory and experiment in acoustics that excited acoustics researchers. (Projects like this also benefited from the insights of psychologists. As it turned out, projects in noise reduction or concert hall design often turned out to be more about “perceived” decibels rather than actual decibel measurements.)
As all of this was happening, Beranek gradually reduced his teaching load at MIT — to 75% in 1951, 50% in 1953, etc. He resigned his tenured professorship in 1958. More and more work — from government contracts, private industry, and research grants — steadily rolled in. As the business grew and contracts rolled in, the MIT and Harvard research communities provided an ample supply of part-time minds and hands. Sometimes individuals were brought in because they were acutely needed. In other cases, the firm simply saw a talented individual and would find some excuse to bring them on.
As the business kept expanding, in size and scope, Beranek set his sights on building up more of a talent base in computing. During the war, Beranek remembered how much of the work of his Harvard Electro-Acoustic Laboratory and its partner lab, the Psycho-Acoustic Laboratory, related to problems dealing with information handling in warships. BBN already combined psychological insights in its acoustics work, working on problems like pilots’ ability to process noise in cockpits. Expanding into the arena of man-machine systems felt natural. To Beranek, this felt like a fascinating area of research with the potential to amplify human labor.
BBN enters the computing business
Beranek’s motto in making hires at the firm was, “Each new person hired should raise the average level of competence of the firm.” At its core this was a research firm. And in staffing the research firm, he felt it was vital that they only continue to hire people who they felt were as smart as them. With Beranek’s sights now on computing, he seemingly only had one man in mind to help lead the firm’s computing efforts: J.C.R. Licklider.
Licklider had been a young researcher in the Psycho-Acoustics Laboratory at Harvard during the war. Beranek was so impressed with him that, soon after being brought over to MIT, he pushed for Licklider to be hired there as well. Now, he was going to push Licklider to give up his tenured position at MIT to come to BBN. With the promise of generous stock options, the understanding that the firm did cutting-edge research work, and the title of “vice president in charge of man-machine and information systems,” Licklider came aboard in 1957.
One co-worker said of Licklider’s move that, “he was becoming entranced by computers at this point and felt he could pursue these interests best at BBN.” In the end, Licklider would go on to draw many of his former friends and colleagues from graduate school, the MIT faculty, and elsewhere in his academic travels to come to BBN. After moving to BBN, some simply continued with their pre-existing work under research contracts they’d won before joining BBN that paid for their services. Some were folded into BBN’s regular operations. Others found contracts to do odd things like write textbooks using BBN’s rare equipment.
But before this collecting of colleagues started, it was just Licklider joining the firm from his MIT position. No formal roadmap had been agreed upon with BBN management. Licklider quickly got things rolling, though. Beranek describes Licklider’s early tenure at the firm as follows:
Lick, as he insisted that we call him, was outgoing and always on the verge of a smile; he ended almost every second sentence with a slight chuckle, as though he had just made a humorous statement. He walked with a gentle step, often with a Coca-Cola in hand, and he always found the time to listen to new ideas. Relaxed and self-deprecating, Lick merged easily with the talent already at BBN. He and I worked together especially well: I cannot remember a time when we disagreed.
Licklider had been on staff only a few months when he told me, in the fall of 1957, that he wanted BBN to buy a digital computer for his group. When I pointed out that we already had a punched-card computer in the financial department and several analog computers in the experimental psychology group, he replied that they did not interest him. He wanted a then state-of-the-art digital machine produced by the Royal McBee Co., a subsidiary of Royal Typewriter.
“What will it cost?” I asked.
“Around $30,000.” (~$300,000 today) he replied, rather blandly, and noted that this price tag was a discount he had already negotiated.
I exclaimed, “BBN has never spent anything approaching that amount on a single research apparatus. What are you going to do with it?”
“I don’t know,” Lick responded, “but if BBN is going to be an important company in the future, it must be in computers.”
Although I hesitated at first — $30,000 for a computer with no apparent use seemed just too reckless — I had a great deal of faith in Lick’s convictions and finally agreed that BBN should risk the funds. I presented his request to Labate and Baruch, and with their approval, Lick brought BBN into the digital era. Lick sat at that computer many hours each day, literally hoarding the machine, learning how to do digital programming.
To be sure, this turn of events was probably not surprising to Beranek. Licklider had been deeply involved with computers at MIT through his affiliation with Lincoln Lab computing projects as well as interactions with Wes Clark and other early computing gurus. It was at MIT where Licklider, as a technically focused psychologist, was becoming an evangelist for work we would now categorize as human-computer interaction. But Licklider may have grown expensive more quickly than Beranek anticipated. As contracts and know-how were gradually building up in the computing area of BBN, the firm bought the first PDP-1 from the Digital Equipment Corporation two years after it purchased the first machine.
After describing the Royal McBee purchase in his chapter of “A Culture of Innovation” — an internal history of BBN by BBN alumni — Beranek described the obvious next step:
Then Lick and I took off for Washington D.C. to seek research contracts that would make use of this machine, which carried a price tag of $150,000 (~$1.5 million today). Our visits to the Department of Education, National Institutes of Health, National Science Foundation, NASA, and the Department of Defense proved Lick’s convictions correct, and we soon secured several important contracts.
When it came to talented researchers and academic engineers, BBN’s instincts as researchers often led the way. The organization was a firm and would not willingly light money on fire to pursue its interests. But if a talented group of researchers and engineers felt a problem area was promising, BBN might spend money to pursue it as long as management felt it could probably earn the money back.
In the case of purchasing the PDP-1, this was a more risky leap than the company usually took. But the risk was quickly mitigated, with BBN quickly selling government contracts in areas like man-machine cockpit system displays and computer-based learning. However, the primary grant driving Licklider’s work on computers at the firm was not a government grant. The grant that proved futuristic enough to support Licklider’s grand vision came from a grant-maker the reader would likely not expect.
Libraries of the Future
The Council on Library Resources was established by the Ford Foundation to study “libraries of the future.” The Ford Foundation was something like the Gates Foundation of its day and, thus, funded a broad swathe of areas. In particular, the libraries of the future contract arose from the growing “information problem” talked about in the research community in the 1960s. Namely, much more work was being published than any researcher could possibly make use of. Licklider and BBN were tapped to explore whether or not modern computing could do something about this problem.
(For those interested, the Council on Library Resources was advised by Warren Weaver, John Pierce, and Richard Bolt in the early stages of its existence. Weaver and Pierce have been featured in earlier pieces on this Substack.)
Contracts like this two-year contract were not likely where BBN made the lion’s share of its money. However, projects such as this — or projects such as those where NASA paid the company a fee to perform exploration and produce a textbook — allowed BBN to offset the costs of explorations its researchers already wanted to do.
Going above and beyond what the Council on Library Resources likely expected, Licklider’s team used this contract as an excuse to explore seemingly futurist technology. For example, Licklider and the project team dedicated much of the second half of the report to technical explorations under the assumption that future library-goers might interact with the library as a store of knowledge with a question-answering front end, rather than as a means to check out books.
The project seems to have gone a long way in offsetting the costs in a variety of explorations by BBN staff. Below, BBN-alum John Swets describes a question Fisher Black — of Black-Scholes Model fame — pursued:
Fischer Black, a mathematics graduate student at Harvard and part-time BBN employee, produced a series of question-answering systems that involved symbolic logic and computer programming, including one in which a non-human system first solved the “airport problem” posed by McCarthy. Based on some statements about a person’s whereabouts, transportation resources, and local geography, the system answers the question of how to drive to the airport (in part: walk from the desk to my garage, drive my car to the airport.)
(Eventual Turing Award winners John McCarthy and Marvin Minsky were also part-time members of the Libraries of the Future project team.)
Other ideas explored in the report include:
The importance of improved random access memories for optimal information retrieval on large datasets — like one containing the entirety of scientific papers.
Black’s question-answering system which successfully used first-order predicate calculus to represent information.1
Several prototype indexing methods and iterative information retrieval methods for searching academic papers.
And many qualitative aspects of PC user experience which Licklider is known for.
Many ideas in the report were not novel on their own. However, many of the systems outlined in the report were novel, practical advances that both fulfilled BBN’s contract obligations and the research ambitions of its employees. The report paints a remarkably prescient picture of how personal computers would eventually organize knowledge and interact with users. In reading the report, one can easily recognize many of the ideas for which Licklider would later become famous as the PC revolution got underway.
In this period, BBN was also working on more engineering-focused computing projects. The practical experience and staff the firm acquired in undertaking these projects gave the firm the much-needed experience and talent it would later need to complete the ARPAnet project. The most important of these projects may have been BBN’s work developing small machine time-sharing systems for hospitals.
BBN’s Hospital Time-Sharing Systems
In 1958, Ed Fredkin had only briefly been at Lincoln Lab before deciding to go into business for himself. To get started, he ordered himself a Royal McBee. He did not yet have the money to pay for the computer, so he went seeking out contracts. He consulted with his friends Frank Heart, then at Lincoln Lab, and Tom Marill, who had recently left Lincoln Lab for BBN, to see if they knew of any customers. Marill told Fredkin to see if BBN would be interested in doing some business with his company. So, he set a meeting with BBN’s new computing czar, J.C.R. Licklider. Fredkin’s meeting with the newly-joined Licklider ended with Licklider pitching Fredkin to, “come work at BBN.” Licklider convinced the partners to purchase Fredkin’s Royal McBee as part of his hiring arrangement. With that, Fredkin was added to Licklider’s growing team of computer engineers.
Licklider’s computing team buildup was still in its early stages, as Fredkin remembers it. If you said “computer,” people would say “analog or digital?” to show they at least knew something about computers. In Fredkin’s estimation, even Licklider was very much still learning the ropes. It was clear that Lick’s higher level ideas were fantastic. But as an engineer, Fredkin had this to say about him:
There was nothing I could do to get Lick to be a good programmer. He insisted on being a coder, and had wonderful high-level ideas; but what he always chose to code never made sense to me. I tried to straighten him out a number of times but couldn’t succeed. It’s just that at that time he didn’t have a knack for coding, either in the style of code or in the things he chose to code.
In short, in Fredkin’s estimation, Licklider seemed to be more bird than frog at that time.
But Fredkin may have had high standards. Licklider referred to Fredkin as a “young genius” and said the decision to bring him in to help grow the computing efforts was a no-brainer. Fredkin, who at the time was not the most fit for a corporate environment, found Licklider and BBN very willing to find a way to make the arrangement work. Licklider remembered that “he was having all kinds of psychological problems about getting work done.” So, BBN worked out an arrangement with Fredkin where, at the end of the month, they would figure out what his salary would be for the month. To Lick, this was just one of BBN’s many “fantastically interesting and flexible arrangements” that it worked out with its sharp employees.
Fredkin was instrumental in convincing BBN to buy the first PDP-1 computer and to bring on professors Marvin Minsky and John McCarthy as part-time employees. Soon after this arrangement began, Fredkin surprised McCarthy by saying that he believed some of McCarthy’s time-sharing concepts could actually be implemented on the rather small PDP-1. Fredkin’s combination of engineering design skills and professional connections from Lincoln Lab helped push this work forward. As McCarthy remembers:
Fredkin designed the architecture of an interrupt system and designed a control system for the drum to permit it to be used in a very efficient swapping mode. He convinced Ben Gurley, the chief engineer for D.E.C, to build this equipment.
Fredkin and his colleagues made multiple technical improvements to prove McCarthy’s ideas could be implemented on such a small machine — including inventing the concept of a swapping drum. Fredkin left the firm in 1961, with McCarthy taking over directing some of Fredkin’s time-sharing projects. But Fredkin’s efforts did a lot to jumpstart BBN’s engineering efforts in time-sharing.
This early time-sharing work eventually led to BBN being hand-picked for an extensive course of funded engineering research. The work came about when Jordan Baruch, one of BBN’s partners, was doing acoustics work for the Clinical Center at the NIH. His project involved instrumentation for the cardiac and neurological staff. During his weekly trips to the NIH, Baruch would go to the home of the Clinical Center’s Director, Jack Masur, for gin and jelly beans. On one of these evening visits the topic of computing in healthcare came up because Masur was asked to give a speech on the topic. Baruch seized the opportunity and shared his vision for the possibilities of computers with patient medical records. Masur used many of Baruch’s ideas in his speech. After the speech was a hit, Masur phoned Baruch and told him, “You should go do it.” Masur recommended Baruch apply for a grant from the NIH’s Division of General Medicine to build the system. BBN did. It won the $1 million (~$10 million today), three-year initial grant.
The contracting relationship would continue over the years, growing from a proof-of-concept prototype to an actual system deployed in a hospital. The specific goal of the research project was for BBN to develop a hospital computer system that would automate “the information gathering, processing, storage and retrieval that takes place in the modern hospital.” The work eventually grew into BBN installing terminals around Massachusetts General Hospital (MGH) which were connected to a time-shared computer at BBN. The initial applications were admissions, discharge, medication ordering, test ordering, and test reporting.
As the project wore on, the medical professors at MGH began using the system for certain types of interactive (albeit primitive) data analysis for their medical studies. BBN staff worked with them to develop these capabilities in the system. As former project manager Paul Castelman noted, “At the 1965 RAND/SDC conference on advanced data-management systems, BBN’s Hospital Research System was the only system reported that operated interactively rather than in batch mode.”
Four years after the start of the MGH project, work on the project began slowly winding down. As a technological proof-of-concept, the project was a success. But it seemed that the system’s deployment in a teaching hospital found difficult reception with some of its (apparently many) opinionated faculty. As project member Bill Mann remembered:
The feelings at MGH ranged from “very interesting” to “it may kill my patients, get it out of here,” with a strong bias towards the latter.
In the mid-1960s, the project’s originator, Jordan Baruch, was leaving BBN to spin off a (BBN-sanctioned) joint venture with GE based on the work called Medinet. The MGH system itself was being handed over to the hospital to operate. In this period, Frank Heart joined BBN as a Vice President in charge of managing technical engineering, with a particular interest in BBN’s life sciences efforts. Heart, in his first year and a half at the firm, helped spearhead BBN’s life sciences computation efforts in various areas of biomedical research.
However, not long after Heart’s joining word was swirling around that Larry Roberts had been brought to ARPA to orchestrate a contract that, as an engineering problem, was too fun for the team at BBN to pass up. Heart and several other Lincoln Lab alumni who had joined BBN in this same period were ideally suited to carry out Roberts’ project. They had a toolkit few in the computing research ecosystem had at the time: deep experience in designing and implementing real-time systems.
ARPAnet Context and Beginnings
The computing team that had been amassing at BBN contained several individuals from the small community specializing in real-time computing systems. In his oral history, Frank Heart explained why so few in the computing community had real-time systems experience, saying:
You know, in that period of time, the computer world was really somewhat divided into people who understood real-time systems and those who didn't. I mean, there were people who built computer systems using operating systems, and there were other people who built very finely tuned machine language programs for dealing with phone lines. And those two camps didn't interact a great deal. So the world was not full of people who knew how to make computers run in real-time, and connect to real-time systems. We were not the only group, but it was a somewhat small universe.
Making computers work via communication lines was no easy task. Larry Roberts, the IPTO PM who oversaw the ARPAnet project, had his own difficulties with the technology while at Lincoln Labs. In his last round of research at Lincoln Labs before joining ARPA, Roberts had been attempting to outline the open computer and communication issues that stood in the way of achieving practical inter-computer communications. As Roberts remembered, the results of this work he’d done with Tom Marill showed “that computers could work with each other, and we figured out how to do that — but we couldn’t get the communications to work on it at all.”
This problem, inter-computer networking, is the primary problem Roberts was brought to ARPA by IPTO Director Robert Taylor to work on. Roberts, who initially rejected Taylor’s offer since he so enjoyed his life as a researcher at Lincoln Lab, eventually had his arm twisted by Lincoln Lab staff. He had turned down Robert Taylor’s offer in early 1966. But, after failing to find another candidate as good as Roberts, Taylor went with something of a Plan B. Then-ARPA Director Charles Herzfeld, at Taylor’s bidding, called up Lincoln Lab’s Director to point out that ARPA was 51% of Lincoln’s funding. Herzfeld then re-emphasized just how much they would like to have Roberts’ services. Roberts accepted. Or, as Taylor summarized these events, “I blackmailed Larry Roberts into fame!”
For reasons beyond the diet blackmail, Roberts was coming around to what he viewed as a bit of a research management job. Roberts explained:
I was also coming to the point of view…that this research that we [the computing group at Lincoln] were doing was not getting to the rest of the world; that no matter what we did we could not get it known. It was part of that concept of building the network: how do we build a network to get it distributed so that people could use whatever was done? I was feeling that we were now probably twenty years ahead of what anybody was going to use and still there was no path for them to pick up…So I really was feeling a pull towards getting more into the real world, rather than remain in that sort of ivory tower.
This general sentiment, coupled with BBN’s openness to interesting research problems if the money was right, is partially why the firm was so successful in recruiting Lincoln Lab staff.
Roberts’ willingness to dedicate the next years of his career to the specific problem of orchestrating the building of the ARPAnet is not surprising. He had started heavily thinking about the problem four years prior. The event that pushed Roberts to begin dedicating his research agenda to the idea was a series of conversations he had at a 1962 computing conference with, among others, J.C.R. Licklider. Roberts said this of the ideas that came up talking with the group:
I came to the conclusion that the next thing, really, was making all of this incompatible work compatible with some sort of networking. In other words, we had all of these people doing different things everywhere, and they were all not sharing their research very well. So you could not use anything anybody else did. Everything I did was useless to the rest of the world, because it was on the TX-2 and it was a unique machine. So unless the software was transportable, the only thing it was useful for was written technical papers, which was a very slow process. So, what I concluded was that we had to do something about communications, and that really, the idea of the galactic network that Lick talked about, probably more than anybody, was something that we had to start seriously thinking about. So in a way networking grew out of Lick's talking about that, although Lick himself could not make anything happen because it was too early when he talked about it. But he did convince me it was important.
According to Robert Taylor, the recent development of time-sharing was the reason he felt his tenure as IPTO chief was a particularly good time to make a push for the ARPAnet project. Taylor said:
The thing that struck me about the time-sharing experience was that before there was a time-sharing system, let's say at MIT, then there were a lot of individual people who didn't know each other who were interested in computing in one way or another, and who were doing whatever they could, however they could. As soon as the time-sharing system became usable, these people began to know one another, share a lot of information, and ask of one another, "How do I use this? Where do I find that?" It was really phenomenal to see this computer become a medium that stimulated the formation of a human community.
The ARPAnet, in Taylor’s mind, was meant to make this “interactive notion” work one level higher — across universities rather than just within them.
Prior to Roberts’ arrival, Taylor had already gotten the necessary funds for the early stages of the project approved and set aside by Director Charles Herzfeld. The next step in the project is common to many major ARPA projects like this. Larry Roberts went around talking to anybody in the ARPA computing community who could conceivably be useful. He wanted to have an understanding of all the opinions from all the top people before he put a concrete proposal out into the world.
Many researchers and engineers contributed ideas that made it into the final proposal, such ideas to improve the network’s host-to-host protocols. But one researcher who was consulted in the planning stages of the RFP can be considered of particular importance. That researcher was Wes Clark of Lincoln Labs — who mentored several BBN employees while they were there. Clark, who his contemporaries considered a bit of an inventive genius, is a bit of an unsung hero of this generation of computing for his work in building early small computers — such as the TX-0, TX-2, and LINC. Clark’s thoughts helped convince Roberts that instead of using a large machine in the center of the country to help “run” the ARPAnet, Roberts should have a small computer at each site handle that. Clark’s advice was singularly important in determining the final structure the ARPAnet took — which was remarkably decentralized.
These small computers Clark proposed came to be known as “IMPs.” In the ARPAnet, the IMPs were meant to connect to both the large host computers at the ARPAnet site as well as the corresponding IMPs at other ARPAnet sites. If they could be successfully made, the use of IMPs would substantially ease the burden involved in connecting host sites to communication lines. Instead of each site having to find a way to connect directly to the telephone lines, they’d simply connect to an IMP. It would be the main contractor’s job to make an IMP reliable and easy enough to use to make the ARPAnet work.
Issued in 1968, the RFP was seeking a prime contractor to design, build, and install these IMPs. In essence, whoever won the contract would be principally in charge of the ultimate success or failure of the project. The ARPAnet nodes — largely research sites with a lot of IPTO funding already — would receive their own ARPA contracts to perform some amount of set-up, debugging, and research on their own end. The IMP contractor would be in charge of building IMPs that would be able to interface with most of the large computers at the time — which were not built to be interoperable. This IMP contractor also had to make sure that IMPs could communicate across noisy telephone lines, across thousands of miles, in real-time. This was the problem that had stumped Larry Roberts on a small scale while he was still at Lincoln Lab. This was a difficult research project as well as an intricate kind of engineering implementation problem.
The team at BBN thought this sounded like great fun. Each member of the proposal team — Frank Heart and Severo Ornstein on hardware, Dave Walden and William Crowther on software, and Robert Kahn on theory — make a point of repeatedly emphasizing that the project just seemed to be great “fun” and a fascinating engineering puzzle to solve. One or two felt it was a problem of massive importance. More often, they did not quite grasp how significant the project would become. Regardless, it was clear that this was a contract BBN wanted in on. So, Frank Heart and Robert Kahn — who had been consulted during the RFP process by Larry Roberts — took the lead in BBN’s proposal efforts. Kahn spearheaded the conceptual aspects of the proposal. Heart, who would eventually be put in charge of the project, spearheaded the more practical engineering aspects.
Kahn, who would later become IPTO chief, had also found his way to BBN around the same time Heart had. In 1966, Kahn took leave from MIT and joined BBN on the advice of a fellow engineering professor. Kahn’s research focus on mathematical aspects of communications was very theoretical. Despite Kahn having worked stints at Bell Labs during his Princeton Ph.D., he was still driven mostly by theoretical problems. So, Kahn’s senior colleague felt he would benefit from additional practical experience. Kahn referred to this piece of advice as “the best advice I ever got.” So, he headed to BBN. He initially intended to be there for only a year or two — he ended up being there for around six years. At BBN, he chose to work on networking.
He described his intentions as to “get my hands a little more dirty with some of the practical problems of everyday engineering.” He seemingly felt BBN was the best place to do this. He described BBN as follows:
It was a marvelous little firm at the time, a fraction of the size it is today. It was largely a group of very professional people doing an extension of what Harvard and MIT were doing, except on a full-time basis, without the responsibility of dealing with students and teaching.
With the RFP out, BBN put far more person time into the proposal than was normal for the firm. Five or six full-time people worked “very, very long hours” on the project for approximately three to six months. As Severo Ornstein remembered, “I spent a lot of time, I remember nights till 3 or 4 in the morning, working with Bob Kahn in the back room of my house in Newton on the proposal — designing the system and figuring out how it was all going to work.” Ornstein also noted that, “more dollars were spent preparing that proposal, more man hours charged to it, than I think had ever been done for any [BBN] project.”
As a reference, the firm strove for 70% of an individual’s time to be billable to some contract in some way. This inordinate use of time on a single proposal was a special case. To be sure, the special case was probably partially driven by the understanding that the contract could lead to longer-term contract extensions. But it seems that the primary reason the team spent as much time as they did was because both BBN employees and management believed this to be an exciting technical problem area — one which the firm felt they knew a lot about.
Many researchers on the team had known Larry Roberts at Lincoln Lab, knew him to be exceptionally smart, and figured he would understand how important their real-time computing experience was. Heart’s Lincoln Lab’s work with the Whirlwind computer required the processing of radar data in real time. As he put it, this meant “the computer had to accept the data at phone line rates and deal with each radar scan before the next one came along.” Heart continues, describing what he believed made his BBN engineering team so attractive in his chapter of “A Culture of Innovation,” saying:
This kind of computer use was unusual at the time, but at Lincoln Laboratory the group of people working with me became unusually expert at the real-time use of computers. At Lincoln, in a long series of projects, computers were connected to various radar antenna systems, radio antenna systems, sensors at underground seismic arrays, and sensors at underwater acoustic arrays. Each such project required a detailed understanding of the computer timing relative to the sequence of data arriving from the various sensor systems. This experience at Lincoln — tying computers to phone lines, and constructing hardware and computer programs that involved the timing constraints of such data handling — was a crucial attribute of my group at BBN that years later bid on and won the ARPAnet contract.
BBN’s main competition for the contract was large contractors like Raytheon and Western Union. Companies like this were significantly less focused on cutting-edge research than BBN but had deep experience managing large systems integration and implementation projects. Roberts stated in an oral history that, on paper, he thought that Raytheon’s bid looked quite reasonable. It wasn’t in a completely inferior league to BBN’s. But Roberts, himself having come up at Lincoln Labs with many BBN staff, did not ignore the advantages that the firm’s structure offered. He recalls:
Raytheon had a good proposal that competed equally with BBN, and the only distinguishing thing in the long run for my final decision was that BBN had a tighter team organized in a way that I thought would be more effective than a very steep commercial structure with lots of managers and vertically – so I didn’t believe they’d keep the schedules as well, but they had a good proposal.
To be clear, ARPA often trusted Raytheon with a variety of implementation contracts. So, it is not likely that Roberts meant that Raytheon would not reliably stick to an implementation schedule. Rather, he likely meant that BBN was far more likely to solve its way through all of the novel engineering challenges that stood between these proposals and an actual system being implemented. This was an extremely novel system. That was likely why BBN won the day.
Of course, all engineering projects are at least, in part, research projects; they all have many “unknown unknowns,” even once they are solved on paper or have been implemented before. Those standard challenges obviously existed in this project. However, the ARPAnet system had so many major “known unknowns” that this was as much a research project as an implementation project. The whole system was an experiment. In fact, the research-focused Robert Kahn stayed on with BBN for three more years after BBN won the contract because of this. There were practical, ongoing research problems — such as those in the area of systems design — that were going to require continuous re-working as the project progressed. As Kahn explained:
My notion originally was "Help them get the award; let them go build it and I will go back to doing what I was doing." But I pretty quickly came to realize that that just wasn't a practical notion. There were too many things that had to be thought about.
According to Roberts, companies like IBM even ‘no bid’ the contract because they didn’t think the request, as it was written, was possible. The price for the computing tasks required, they thought, was completely infeasible with the technology that existed. Roberts remembered:
IBM and CDC 'no bid' it because, they told me: "If we used Model 50s at these sites, you're going to go broke, and we think you're crazy." And what else is there, of course, besides Model 50s?
Many corporate researchers in the communications industry felt that the packet switching idea, which the network’s IMP-IMP communications relied upon, was completely infeasible. To players like AT&T, this was an academic idea that head-in-the-clouds researchers came up with; this was all fine and good on paper, but would just not hold up to the practical engineering reality. BBN obviously disagreed. But, as Frank Heart put it, “There was no proof. You couldn’t go to some book and find the answer; it was an experiment.”
The challenge, which some thought impossible at ARPA’s price point and given the noisiness of telephone lines, was set. BBN won the IMP contract. It was now their job to make it all work.
ARPAnet Operations
The initial one-year contract Roberts awarded BBN was to develop the IMP and deliver a four-node network connecting UCLA, the Stanford Research Institute, UC Santa Barbara, and the University of Utah. The contract was for about $1 million in 1969 (~$8 million today). The initial team that worked on the contract that year was made up of the initial five members who drew up the proposal along with about three more full-timers, primarily to work on hardware, and four or so additional part-timers. Several of the hires were Severo Ornstein’s best students from his course at Harvard.
The proposal team had, at least conceptually, worked its way through many problems in the extensive time it spent working on the proposal. So, the team was able to hit the ground running and make a push to have the four-node network operational by the end of the year. In all likelihood, Roberts likely expected the team might not make the deadline. In many cases, ARPA contracts would set extremely ambitious benchmarks that PMs, office directors, and contractors would later re-negotiate. But this did not turn out to be one of those cases.
Other Contractors on the Project
A number of subcontractors were also involved in the ARPAnet project. Each of the node engineering departments was a subcontractor. To a certain extent, these departments were responsible for getting their host computer set up with the IMP. BBN would do what it could to make this as easy as possible for them. But given the state of the technology and the talent present at the nodes, this was deemed to be the optimal approach. Additionally, ARPA used a contractor called DECCO to take charge of the logistically annoying process of purchasing time from AT&T on their telephone communication lines.2 Of course, DECCO purchased what BBN told them to purchase. Only BBN knew what it needed to successfully hook the IMPs up to the phone lines. But one would imagine BBN was still thankful to have DECCO to keep them somewhat insulated from this process.3
Two other contractors of note were the Network Analysis Corporation and the Network Measurement Center. ARPA wanted the Network Analysis Corporation to run simulations to help design optimal network topologies. In theory, this was reasonable. But, in Frank Heart’s estimation, the company did not prove as useful as ARPA expected in determining which sites to add next. In his eyes, practical matters such as whether or not departments were properly staffed or were ready to work on their IMP were mostly responsible for determining the network’s shape as it grew. The Network Measurement Center was run by Leonard Kleinrock at UCLA — the first node of the ARPAnet — and was responsible for taking a variety of measurements to study the network. The Network Measurement Center was, to Heart, somewhat academic in nature and not pivotal in the initial engineering or design of the ARPAnet. But it was useful in taking measurements pertaining to the health and efficiency of the network in its early stages.
Lastly, Honeywell was the subcontractor most pivotal to the contract’s success. BBN had no manufacturing facilities. Thus, they needed to contract with a computer manufacturer to have an IMP of their design actually built. At the time, few computer manufacturers did this. A standard company’s manufacturing process was not built to do wholly custom jobs. The processes were suited to be fiddled with, but not much more than that. But Honeywell had recently acquired a company called the Computer Control Corporation (CCC) that made them an exception. The team from CCC, as Heart put it, “understood a bit about the special systems business.” This was a unique enough skill that at least two contract bidders had used Honeywell as a subcontractor — BBN and Raytheon. Larry Roberts listed this as a reason that Raytheon’s bid was being considered alongside BBN’s.
Some of the Major Technical Problems
A variety of technical problems awaited the BBN team.
One of the major early problems the BBN team spent its time working through was the error control and correction problem. To work through this, the team was able to design special checksumming hardware that would be installed at both the transmitting and receiving end of the circuit. Every packet of information sent would have a 24-bit cyclic checksum appended to it which would be recomputed by the hardware at the receiving end to determine if there were any errors. If there was, the packet would be retransmitted.
How the nodes got there was the most theoretically-involved piece of this engineering experiment. The packet switching technologies we know, love, and hyper-optimize today were much more un-tested back then. This was one of two areas in which Robert Kahn’s presence proved extremely useful to the engineering-focused team. Kahn was able to quickly explain to them various aspects of packet switching algorithms as problems came up. (The other major theoretical area Kahn contributed to was in some mathematical aspects of phone line error problems.) The technical details of the routing algorithms the team initially deployed are described in the project’s completion report to ARPA as follows:
The approach taken was to use a distributed adaptive traffic routing algorithm which estimates on the basis of information from adjacent nodes the globally proper instantaneous path for each message in the face of varying input traffic loads and local line or node failures. Each IMP keeps tables containing an estimate of the output circuit corresponding to the minimum delay path to each other IMP, and a corresponding estimate of the delay. Periodically, each IMP sends its current routing estimates to its neighbors; whenever an IMP receives such a message it updates its internal estimates. Thus, information about changing conditions is regularly propagated throughout the network, and whenever a packet of traffic must be placed on the queue for an output circuit, the IMP uses its latest estimate of the best path.
Another major area of problems came from the technical aspects of the IMP-Host interfaces. The engineers spent substantial time hitting their heads against the wall to find ways to successfully connect IMPs to all different kinds of computers while, also, minimizing the amount of engineering work required by the node contractors to connect their machine to the IMP.
Solutions to all of these problems had to work in close to real-time. Individuals from remote machines were going to use the network to do much more than just send batch messages like emails. They were going to use their computers to remote into another computer as if the whole network was a single, time-shared machine. So, to make this all work at time latencies that made sense, the BBN team had to do things like “write very carefully tailored programs in machine language to optimize the capacity and the low delay of the data path in the switching node.” The team emphasized in its final report that, “Great attention was paid to minimizing the operating time of the inner loops of these programs.” Many areas of minutiae were all-important.
Many problems — from IMP design, to Host-IMP connection software and hardware, to error-correcting software — required this level of care. All of the high-level concepts — computers sending a message across a telephone line, linking two computers together via a communication line in a lab setting, basic message handling across noisy communications lines, etc. — had been demonstrated as feasible before. But BBN had to invent and engineer all that was required to make it go from technically possible to actually usable.
Viewed naively, the implemented packet switching technology was the team’s major “new” contribution. But, as most engineers will tell you, a project like this can be one big pile of unknown unknowns. That, in part, is probably why industry did not consider touching it in their R&D efforts. But, given the BBN staff’s continuous experience in recent years working on real-world time-sharing systems and real-time computer communications, they were probably the team that had the best sense of what the unknowns of the project might look like and how to deal with them as they came up.
The Honeywell Partnership
To get the IMP designed and built, the BBN team heavily leaned on Severo Ornstein. Ornstein had previously worked with Frank Heart and others on various computer projects at Lincoln Lab, worked with Wes Clark on smaller computers at WUSTL, and had since returned to Cambridge to work at BBN and do some lecturing at Harvard. Prior to the networking project, at BBN Ornstein had been working on projects related to the use of computers in education. Ornstein’s interest in the ARPAnet RFP caused him to dedicate more of his time to BBN.
While designing the IMP during the proposal process, Ornstein estimated that he and the team “did 90% of the design then.” But even when BBN had won the proposal, Ornstein still had to put substantial effort into working with Honeywell to get the computers made. Ornstein remembered:
The man from Honeywell who was assigned to build those interfaces from my drawings didn’t understand the drawings well and was not really careful. We ended up having to redo much of his work.
Much of BBN’s interactions with Honeywell may cast some doubt as to whether an industry actor could have carried out this contract nearly as effectively as BBN had. One account by Ornstein regarding the BBN team working their way through a confusing problem will help elucidate why exactly this is. The story both shines some light on what exactly the BBNers were spending their research time on prior to the implementation stages and how Honeywell dealt with problems. Ornstein explains:
They were not used to doing some of the kinds of things we were doing…For example, we discovered some design flaws in the machine. One of the reasons we chose the Honeywell 516 was that we thought it was a mature machine…that was not going to gives us grief. Well, we were wrong. We pushed it very hard…and we uncovered a bug that they could hardly believe, a synchronizer problem…Synchronizer problems are very, very subtle. We had to dig and dig and dig at them, and finally from their back room they produced a really smart enough guy — they do have a few — but, it was very hard to get this guy. We finally got him…when he came and we sat down and I finally had someone I could talk to who would understand what I was talking about and believe me. It was a subtle problem. The program would run fine in the machine for days on end, and at the end of three or four days suddenly the machine would just die, inexplicably. It was a very, very low frequency failure, so infrequent that you could never look at it on a scope; you could only see the effects afterwards. In fact, we had to build special hardware that beat on the trouble spot many, many times faster than normal usage would. Then finally, with all the lights out in the room, we could faintly see on the tracing scope an occasional failure. That was when the Honeywell people finally became convinced that there was a real problem…We showed them the trivial fix they could make to the machine…So, their people weren't absolutely top-notch people; they were okay. They were industrial-strength people, not research-strength people.
Of course, there were industrial operations with plenty of “research-strength” individuals back then — assuming Ornstein meant individuals with the brains and toolkits to consistently find answers to vexing problems like the one mentioned. Bell Labs, Kahn’s old employer, was just one example.
Not assigning this systems engineering-style contract to an academic department makes sense. Most academic departments are just not staffed and structured in a way to make projects like this feasible. But it’s unclear how an outfit like Raytheon would have done working through problems like the one described above. Would they have been able to solve them? If so, how quickly? We see that the Honeywell team was great at reproducing duplicate machines once they had already made one. But there were certain computer engineering principles and skills utilized in the project that some of the builders — from places like Lincoln Lab and Wes Clark’s WUSTL team — had much more experience with than most other groups. That was to BBN’s great advantage.
All of that being said, the formerly-university-employed engineering team still conceptualized this as more of a novel engineering problem than a research problem. Dave Walden, one of the project’s two founding software engineers who later became General Manager of BBN, described how the team viewed the problem as follows:
My view is that mainly what we were doing was we were very pragmatically doing engineering. We had to send these bits down the wire: how do you put a header on the front; how do you put a trailer on the back. There was theory of how you put error-correcting codes on it. Bob Kahn knew that theory and told us what it was. There were some constraints: this is the way that the 303 (or the 301 or whatever the Bell modem is) has to be interfaced to, but after that it was all pretty pragmatic. Not lots of theory of coming from someplace else.
The Team’s Management
Before diving into how the one-year contract’s implementation went, we should first get a better understanding of how the team was managed at BBN. Firstly, it was much smaller than one would expect.
As the first year of work progressed, the team remained small — about eight people. This is how Frank Heart preferred it. As he saw it, that small team of elite talent in the relevant sub-areas was all BBN needed. He described the project’s work style further, saying:
I think mostly I tend to believe important things get done by small groups of people who all know all about the whole project. That is, in those days all the software people knew something about hardware, and all the hardware people programmed. It wasn't a group of unconnected people. It was a set of people who all knew a lot about the whole project. I consider that pretty important in anything very big. So I suppose if you call it a management style, that would be something I'd state. I think also that they were a very, very unusually talented group. I think things tend to get done best by small groups of very, very good people — if you can possibly manage that. You can't always manage it. So if you again want to call it a management style, it is to get the very, very best people and in small numbers, so they can all know what they're all doing.
Heart was extremely averse to the team growing so large that communication had to be done on paper. He preferred their style of “very, very frequent interaction on problems.” As Walden remembered, “I don’t remember such a thing as we had a weekly progress meeting. We probably were more in tune with the progress than that; we probably did it hourly.” Heart and the entire, small team did their best to stay technically involved with each other’s work. Heart, as a rule, insisted on technically understanding every bit of the project. The individual team members attempted to do the same, which they felt was quite easy to do. The team all had offices and work stations immediately in the vicinity of each other. As things came up, they’d simply go talk with the relevant individual or call an impromptu meeting.
The Early Node Installations
With some level of difficulty, BBN worked their way through problem after problem like those in the technical problems and Honeywell sections. With a lot of persistence, the team got the IMPs specified in the proposal built and de-bugged. By September of 1969, the hardware and software for the first IMP was ready for installation.
Unlike many ARPA computing contractors, the UCLA department had a core of individuals who were particularly excited about the ARPAnet project. While many of the professors who owned computers saw the ARPAnet as an opportunity for ARPA to force them to further split their computer time, Leonard Kleinrock and others at UCLA had research interests relating to the network itself. Kleinrock, most notably, had written about the theoretical aspects of packet switching in his prior work. Because of this interest, he formed the Network Measurement Center to monitor certain aspects of the network as a subcontractor for the ARPAnet contract.
In early September 1969, BBN sent its first IMP off to UCLA. When shipping the finished IMPs to the host sites, BBN even went as far as to have staff ride alongside the machines on the plane, just in case. For these early sites, Ornstein and some of the software researchers like Dave Walden traveled to the sites to hook the IMP up to the phone lines and work through any setup issues. In the lead-up to BBN sending the machines, the firm shared a set of specs with the host sites so the sites could do as much work as possible beforehand. On the node end this work was usually led by some kind of hardware lead and software lead.
BBN had apparently done quite an effective job of communicating its technology to UCLA. Also, UCLA had apparently done a good job of internalizing the information and building accordingly. According to Leonard Kleinrock, within a day of initially connecting the IMP and UCLA’s SDS Sigma 7, messages were moving back and forth.
This was a massive success. Still, many questions could not be answered until the second installation was done. Only at that point could BBN see what happened when two nodes tried to communicate with each other. The next month, the BBN team installed its second node at the Stanford Research Institute — which had also been subcontracted by ARPA to do some research on the network itself. Once BBN and SRI had the IMP hooked up, the two node research teams attempted to send the first message over the ARPAnet. As Leonard Kleinrock remembers:
On October 29th, 1969, one of my programmers, Charley Kline, and I were in this room and we decided to log on to this [SRI’s] machine. Now, to log on, one has to type in “LOG” and the remote machine will type in the “IN” So our job simply was to type in “LOG.” Now, up at the other end, there was another programmer waiting to watch all of this. And we had a telephone connection between these two…so they could talk to each other. What happened is Charley typed the “L” and he asked, “You get the ‘L?’” And the answer was, “Got the ‘L.’” He typed the ‘O.’ “You get the ‘O?’” “Got the ‘O.’” He typed the ‘G.’ “You get the ‘G?’” Wacko! The system crashed. This machine [SRI’s] went down.
So the very first message on the internet ever was “LO,” as in, “Lo and behold!” You couldn’t ask for a better, more effective message.
Surely the BBN team considered it enough of a success that any letters were sent at all. The team proceeded to install a new IMP per month for the rest of the year, making four in total. Problems continuously came up through both BBN’s testing and the node researchers’ testing. BBN fixed the problems as they came up. It does not seem like any of these problems proved particularly devastating.
Kleinrock excitedly described the arrangement as follows:
So we had a four-node network running, and we began to test it and find out what some of the problems were. And we could break this network any time we wanted, we found faults, we had BBN test it, etc.
As problems were worked out, there were regular hardware modifications and software releases that had to be rolled out. The software releases were done, initially, by sending Dave Walden around to each of the sites with his paper tapes to get the software working on the machines. In this way, the first batch of nodes became operational and steadily improved. BBN did this on budget and within the one-year timeline of the contract. Larry Roberts and ARPA were, surely, extremely pleased with the progress. By the end of the year, ARPA contracted BBN to expand the network to 19 nodes.
Before discussing how the project scaled, let’s briefly explore Larry Roberts’ working relationship with the BBN team. Roberts was a very active manager in the ARPAnet project, not just its funder.
Working with Larry Roberts
In putting together the RFP for the ARPAnet, Roberts had looked deeply into many technical points and consulted the best members of the computing community. In doing this, he made several key decisions before the contract was ever in BBN’s hands. As Heart put it, “They [DARPA] picked the baud rate of 50 kilobits, they picked the sites, they picked the issues about the checksums. A lot of the work had been done by DARPA in advance.”
As Dave Walden framed things, BBN could surely be considered the ARPAnet’s operator and the primary driver of the engineering work, but ARPA was the high level manager. This entire operation, the ARPAnet itself, was meant to facilitate new capabilities for ARPAnet contractors. So, Roberts providing a constant guiding hand made sense. Also, the ARPA contracting community contained some of the best minds who could make technical suggestions to the BBN team. So, Roberts and ARPA continually facilitated meetings between BBN, node contractors, and others as they saw fit. The now-famous Network Working Group which debated many ARPAnet-relevant ideas, like what later came to be the TCP/IP protocols, was one example of these efforts.
Once the project was underway, Roberts largely left BBN in charge of making day-to-day decisions. They did not need to seek his approval on many things. But he did keep close tabs with the team and would raise objections from time to time. Roberts could technically “pull rank” on BBN as the project’s funder. But the arrangement did not really work like that. If Roberts didn’t agree with the team’s approach to a particular problem, they would discuss it over the phone or in person until they could come to some kind of agreement. In a contentious case, that might take some number of days. But, in most cases, a clear agreement was reached.
The BBN team held extreme respect for Roberts as an engineer. In turn, Roberts seemed to respect the BBN team greatly. Roberts had a direct line of communication not just with Heart, but with individuals in charge of component technologies on the team such as Severo Ornstein. According to Heart, normally a week wouldn’t go by where Roberts didn’t talk with somebody on the team. Ornstein, who was particularly casual, said he simply conversed with Roberts as he would any other colleague.
Beginning to Scale the Network
The initial contract to build the four-node network was a success. So, ARPA opted to give BBN a contract to spearhead the effort to 19 nodes.
In early 1970, BBN itself became the ARPAnet’s fifth node as the network continued to expand at roughly one node per month.
As the expansion continued, new engineering problems continually arose. While Frank Heart was known to be an extremely defensive engineer, things were still bound to work imperfectly. This whole project was an experiment and was a first-of-its-kind network, so one could only be a certain level of defensive without halting the project’s progress entirely.
In many areas, the BBN team shipped technology that it initially felt was merely good enough — with the understanding that they could continually improve component technologies on an ongoing basis. One case of this was the network’s protocols. The initial NCP protocols were deemed by the Network Working Group to be quite sub-optimal.4 So, over the years the NCP Protocols protocols were improved until they were entirely replaced by Kahn and Cerf’s now-famous TCP/IP protocols.
As the ARPAnet completion report notes, on occasion, some of the problems present in the operational network were more than small. In 1970, a major problem was demonstrated with IMP flow control and storage allocation. Node hosts, if they didn’t heed the error, could accidentally halt the network’s entire operation. Yet, while BBN and others took the many months required to fix the problem, the ARPAnet continued to provide adequate service to its nodes.
As Dave Walden recounted in his oral history:
Walden: …it almost immediately became obvious to anybody who hadn't believed it before that what we had originally implemented was insufficient. And what we had specified to the host and what they had implemented had a lot of ramifications. They were now done and now it didn't work all that well. So for the first year or two years, quite a long time, there were rules, informal rules in the network that you didn't pump stuff into it from the hosts so fast it swamped the net. Because we all knew the algorithms didn't work, I mean they broke, patently they broke when you pumped stuff at them too hard. And so this interim solution worked in that network — this is the kind of thing that somebody who is working on proving programs correct and stuff can't imagine. It actually was quite useful in those early days of experimentation because everyone agreed not to break it. You know how to break it; let's not break it. Meanwhile, let's go figure out how to fix it…
Interviewer: So it was on the order of a gentlemen's agreement. Just not to go break it.
Walden: Engineers' agreement.
Given that all the users were well-trained and capable engineers and researchers themselves, BBN could also get away with only gradually scaling up its service capabilities. Dave Walden running around with his tapes to implement software updates was one example of this. As the network expanded and the network began to operate more reliably, BBN was able to dedicate more of its team’s efforts to helping the network operate a little more automatically.
In these middle years of the project, BBN also began to expand its efforts to finding more efficient solutions to manage network issues, answer questions from nodes, etc. In the beginning, BBN’s answer to this problem was to throw Dave Walden at it. As Walden fondly remembers:
Up until September of '70, I had the program listing beside my telephone at home and would get telephone calls at home whenever something stopped. My telephone number was quite literally on the front of the original packet switches and they called my house over here in Allston and I would talk to whoever was there, using my listings. So that's what happened.
With ARPA sponsorship, BBN began running the Network Control Center (NCC) in service of this goal. In building up the NCC, BBN designed and implemented tools and software to regularly ping the network and keep tabs on network problems, monitor the status of hosts, etc. Alexander McKenzie became the “ARPAnet generalist” BBNer put in charge of the customer service aspects of the project as it grew. McKenzie proudly recounted stories of calling telephone companies and telling them a line was about to go down…and then it would. The companies were quite impressed because this was not an ability that they had yet.
Starting in 1972, BBN hired staff to offer de-bugging and other customer services to network nodes full-time — three people worked the day shift, two the late evening shift, and one the midnight shift. Through BBN’s and the NCC’s efforts, nodes were soon able to (in most cases) automatically report their status. Maintenance and debugging were then typically carried out remotely from the monitoring center. As the NCC’s capabilities grew, BBN was able to successfully place IMPs at locations where the staff had less and less computer training.
Additionally, as the network expanded BBN conducted research in an attempt to expand the types of IMP technology as well as the types of communication lines IMPs could be hooked up to. In 1971, BBN finished designing its first terminal interface message processor (TIP). These terminals were able to connect to the ARPAnet and do work on ARPAnet host machines without having to be run on a host computer of their own. On their own, TIPs were more expensive than IMPs. But the idea was extremely exciting to ARPA because the TIPs becoming operational meant one could hook up to the ARPAnet without having a large computer on hand. The first two TIPs, shown below, were installed at MITRE and NASA’s AMES research facility. The early 1970s saw the first TIPs installed, IMPs connecting to the network via satellite technology, the design of standard IMPs ten times more powerful than the initial ones, and the deployment of Ornstein’s Pluribus multi-processor IMP.
As the ARPAnet hit its 19th node, ARPA had already opted to give BBN a contract to expand the network to over 50 nodes. The first two years of the project had all occurred largely on schedule and on budget. With this next extension, the network even began to include nodes from remote geographic areas like Hawaii and Norway. This required the use of some of the project’s new technological capabilities as well as the project’s first interactions with foreign telephone operators. While these remote nodes did tend to be more buggy, they were largely functional and proved some of the new capabilities to be useful.
Throughout this period, Larry Roberts’ continual arm-twisting of node sites to do what they needed to do to make the node installations a success proved extremely useful in keeping things on track. As many of the sites’ primary computing funder, he could exercise this power quite easily. By 1975, the ARPAnet project was clearly on stable footing. So, responsibility for the project was transferred to the Defense Communications Agency (DCA). According to DARPA Historian Richard Van Atta, at the time the project was transferred to DCA the total budget outlays throughout the project had been $25 million in total (~$150 million today). That number covers the entire project, not just BBN’s portion. By anybody’s metrics, this should be considered a remarkably cheap success.
The First International Conference on Computer Communication
How exactly the outside world — meaning those in the technology community not affiliated with an ARPAnet host site — became aware of the ARPAnet’s success deserves a brief explanation. Even in 1972, many in the communications community doubted this cross-country packet switching network idea would be technically possible anytime soon.
Robert Kahn’s final project as an official member of BBN helped rectify this. In October 1972, he put on a massive demonstration of the ARPAnet at the first-ever International Conference on Computer Communication (ICCC). The conference itself was an extremely useful forcing mechanism for BBN and each of the node contractors. This provided tangible pressure, with a concrete deadline, to thoroughly debug many pieces of technology, sure-up certain protocols, etc. The academics responded much better to the deadlines or threats of their funding being diminished than they did to the mere presence of bugs in the system. Additionally, the existence of the conference provided a push to ensure that as many host sites as possible were functional across as many use cases as possible. To this point, many nodes had only been outfitted with the hardware and software to use some of the ARPAnet’s possible applications.
Dozens of nodes contributed computer hosts to the demonstration. The conference was a sort of DIY showcase where individuals could hook up to the computer hosts via TIPs and work with them the way node users did every day. This could be for things as simple as e-mail, a rapidly growing use case of the users, as well as much more complicated programs. As Roberts recalled:
I think this was the watershed event that made people suddenly realize that packet switching was a real technology. We had thousands of people who went through that — I don't know what the exact number is — at least a thousand who went through that particular exhibit.
Or, as the ARPAnet completion report put it:
The demonstration itself was a spectacular success; with everything working amazingly well, many visitors remarked that the ARPANET technology "really is real."
ARPAnet Results
On a technical level, the ARPAnet experiment was a success in many ways. The project proved that a large network could be built in such a way that node failures were localized and did not crash the rest of the network. It convincingly demonstrated adaptive routing algorithms and packet switching theories. To a large extent, the ARPAnet proved that a decentralized network like this could run itself without a central command center. Finally, from a user perspective, the IMPs and TIPs proved reliable and easy enough to use to facilitate the time-sharing of diverse computers, thousands of miles from one another, for a variety of tasks. The working ARPAnet also succeeded in its role as a test bed for the entire field of computer communications technology research.
But, of course, the success of the ARPAnet project goes beyond the technical successes of the project. As BBN’s ARPAnet completion report stated:
The ARPANET project also proved the feasibility of achieving closely knit communities of technical interest over a widespread geographic area; it is possible that this social feasibility demonstration is as important as the many technical feasibility demonstrations.
Possibly the clearest indicator of this growth is how rapidly internode traffic grew over the network through the mid-1970s.
As the usage of the network grew, BBN feared that even the new algorithms being used to handle routing, flow control, and congestion control would not be able to keep up. But, as the team noted, “Luckily, the improvements in the algorithms managed to stay slightly ahead of the growth in network size and traffic.” In BBN’s estimation, the diminishment in growth rate was because at a certain point, the existing host machines being accessed over the network were simply reaching capacity.
Throughout this growth period, acute problems present in the network lessened, BBN sharpened its processes, demand increased, and more research was being done that relied on the network in some way. Several of the BBNers, in their oral history interviews, talked excitedly about how the network gradually became a “utility.” When prompted by the interviewer to say more, Heart explained:
A utility is something people depend upon. Like the electricity, or the phones, or the lights, or the railroads, or the airplanes. Yes, it was a utility. That's the thing that was the amazing surprise. It was started as an experiment to connect four sites, and it became a utility much, much faster than anybody would have guessed. People began to depend upon it. And that was a problem, because that meant when you changed it, or it had problems, they all got mad. So that was a two-edged sword. But it was also very exciting.
Alexander McKenzie noted that, while the ARPAnet was never as reliable as a power company, the team did eventually succeed in keeping the IMPs up 98% or 99% of the time. As this transition to “utility” took place, NCC-style activities took more of an emphasis. Research and operational activities that optimized the reliability of the network slowly became prioritized over network research activities that created knowledge but interfered with service. This was likely disappointing to some BBNers. But it can also be seen as merely a symptom of the project’s success.
The network was becoming so convenient that a substantial percentage of network traffic was intra-node traffic. Computer hosts at one site were communicating with others in the same location. In other words, communication over the ARPAnet had become, for many tasks, preferable to even walking next door to see a colleague.
In 1972, BBN even began attempting to form a sanctioned spin-off to commercialize the technology in the wider market. ARPA was spiritually supportive of this effort even if they did not fund it. In fact, helping get this firm, Telenet, up and running was Larry Roberts’ first job upon leaving ARPA. He had gone to ARPA to make inter-computer networking a reality. Upon leaving, he was heading into a role where he could attempt to get the technology to as many users as possible. Telenet would go on to have a level of success as a growing network provider — but would not be considered a smashing success. The fact that the ARPA-funded technology showed commercial potential as little six years after the project’s initial funding should also be considered a major success.
If one wanted to nitpick, one could point out that ARPA’s major intended application of the network was as a way to reduce computing costs. This never did become the primary use of the network. As computing costs gradually decreased, this particular use of the ARPAnet became relatively unimportant. The network found better uses.
ARPAnet Lessons Learned (and Caveats)
Many small-scale, PM-relevant lessons are present in this report. As one example, the BBN team maintaining extremely close connections with the academic community while implementing sooner rather than later proved to be a key decision. The academically-driven Network Working Group, with its debates on network protocols in the group’s early years, was at a stalemate regarding what protocols were best. The BBN team could have waited far too long to get the perfect set of protocol recommendations from the group. Instead, BBN implemented the NCP protocols and gradually, with much input from the group, upgraded to the much-improved TCP/IP protocols. Another interesting lesson was Roberts’ use of a demonstration as a forcing function to get all the nodes outfitted with the full suite of ARPAnet capabilities.
Minor learnings aside, it is important that the reader fully appreciate just how important having a contractor like BBN was to the success of the project. Not only was the research and prototyping stage of this difficult project done well and on schedule, but BBN seamlessly transitioned it into a utility. Appreciating why they were able to do this is extremely instructive.
The BBN team was staffed with top minds who cared deeply about the research enterprise but operated within the structure of a firm. Dave Walden, explaining why exactly BBN had earned its nickname as the “third university of Cambridge,” emphasized the following:
BBN provided a support structure for self-motivated researchers who were able and willing to find sponsors for the research they wanted to do. The work was not directed in a top-down fashion; within broad limits, senior researchers were free to pursue their own interests, if they could find the necessary financial backing.
To complement this, BBN also had the necessary staff, experience, and structure to manage successful projects as they grew into larger implementation projects. The firm’s hospital time-sharing work is just one example of BBN managing an implementation project with cutting-edge technology. A project at BBN could start in the deeply exploratory research stage and naturally scale into a utility.
The ARPAnet project showcased exactly what BBN was optimized to do. When given the right kind of problem and the right situation, BBN’s model could help facilitate marvelous feats. The ARPAnet itself, just like Edison’s light bulb 90 years prior, proved to be both a field-changing test bed for scientific experiments as well as a world-changing commercial technology. BBN was ideally suited to make a breakthrough of this nature possible. While the mission creep that comes with a profit motive can often steer a commercial firm off of the course of truly ambitious research, BBN was less susceptible to this in these years. If its research somehow became too derivative, the firm would not have been able to recruit staff anywhere near the competence level it sought.
If every scientific/commercial area had a small research firm like BBN, the scope of what would become possible with an ARPA-like PM’s funding would expand greatly. In one’s role as a PM, if the opportunity to help incentivize or facilitate an organization to operate a little more like BBN, there seem to be few downsides — and massive upside — to do so. The more novel a systems contract becomes, the more suitable it is for a firm like BBN.
BBN created such a compelling, applied environment in which to do research that they convinced many professors to give up tenured positions at MIT to come work at the firm. The firm knew exactly how to make use of these bright, applied researchers. Dave Walden put extreme emphasis on how much the “native wit” of the initial technical team he joined — Frank Heart, Severo Ornstein, and Will Crowther — was required to make the project a success, saying:
They are smart guys. There wasn't much theory in how you build a packet switching network. There was a communications theory, but that was all pretty abstract. One just got out there and did it. All the stuff that is now taught in courses in communications about networks and protocols and all of that, I would say we were mainly (as part of the entire community of the host people) inventing it. The academic analysis tended to come later in a lot of cases.
I’ll conclude with Frank Heart’s fond words describing the organization in “A Culture of Innovation:”
First and foremost, BBN was a great place for a technical person to work, and most people really liked working there. It was a middle ground between academia and the commercial world, with the meritocracy and individual freedom of the academic world, along with the potential reward structure and potential impact on the world of commercial ventures. In the case of the ARPAnet and the subsequent explosion of network activity, it was an extremely unusual opportunity for a technical person to “ride a rocket” of change in the world.”
Thanks for reading:)
If there is demand, I can write a separate piece on how BBN’s culture and management emphasis changed from those described in the piece as the decades wore on. But I considered that exploration out of scope for this particular piece.
Pattern Language Tags:
Utilizing a contractor made up of individuals with research-style goals and training working within a ‘firm’ structure.
Building out an experimental test bed for a research field.
Promoting a coordination/service mechanism to reduce material costs and increase research feedback cycles.
This piece is a part of a FreakTakes series. The goal is to put together a series of administrative histories on specific DARPA projects just as I have done for many industrial R&D labs and other research orgs on FreakTakes. The goal — once I have covered ~20-30 projects — is to put together a larger ‘ARPA Playbook’ which helps individuals such as PMs in ARPA-like orgs navigate the growing catalog of pieces in a way that helps them find what they need to make the best decisions possible. In service of that, I am including in each post a bulleted list of ‘pattern language tags’ that encompass some categories of DARPA project strategies that describe the approaches contained in the piece — which will later be used to organize the ARPA Playbook document. These tags and the piece itself should all be considered in draft form until around the Spring of 2024. In the meantime, please feel free to reach out to me on Twitter or email (egillia3 | at | alumni | dot | stanford | dot | edu) to recommend additions/changes to the tags or the pieces. Also, if you have any ideas for projects from ARPA history — good, bad, or complicated — that would be interesting for me to dive into, please feel free to share them!
Specific Links:
The following oral histories and related materials were a joy to read through and primarily what the detailed ARPAnet portions of the piece were based on. The ARPAnet Completion Report was simply used to fill in the gaps. These give a far more detailed account of how things actually worked on the project than the completion report ever could.
Alexander Mckenzie’s UMN Oral History
Contains many insights on the customer/user service mechanisms BBN utilized as the network grew in the early-to-mid 1970s.
Robert Taylor’s UMN Oral History
Taylor dives into why exactly he felt his tenure as IPTO chief was a good time to attack the network problem and how he went about the early stages of the work.
Leonard Kleinrock explaining early ARPAnet node installation and testing
The History of Telenet and the Commercialization of Packet Switching in the U.S.
For more on the early history of commercial packet switching, check out this IEEE paper in which Larry Roberts is a co-author
Licklider et al.’s Libraries of the Future Report
General Links:
This report was a treasure trove of information related to contracting details, specifics on technical accomplishments, and various graphics summarizing ARPAnet’s growth.
The report also highlights how the actual work deviated from the plan, what was added/removed, etc. as the project went on.
Lastly, the report lists many of the technical problems the BBN team had to work through at various points.
A Culture of Innovation: Insider Accounts of Computing and Life at BBN
Besides being the source of many quotes and pieces of information in the piece, this is the most thorough account of how BBN actually operated on a day to day basis. Each chapter is written by a different, long-time (and often high-ranking) BBN employee.
As one example, the book contains a long section from Frank Heart on how overhead rates impacted project selection and how BBN’s billing system worked. Explanations like this for all sorts of aspects of the operation are included in the book.
The paper is Black 1963.
DECCO was a contracting unit of DCA in communication services
Even though the Bell representative assigned to the project was apparently proactive and quite excellent.
Thanks to Dan Tappan for flagging that it was NCP protocols, not “Telnet protocols.” See his comment as reference
"NCP Protocols" not "Telnet Protocols". NCP was the precursor to TCP/IP as the host-host protocol. "Telnet" is the application layer TTY protocol, that still exists.
https://en.wikipedia.org/wiki/Network_Control_Protocol_(ARPANET)
Great piece about the takeaways for building the research organization of the future - thank you! The history is worth recounting for that perspective. It may be worth noting that another excellent reference is the book "Where Wizards Stay Up Late: The Origins of the Internet" by Katie Hafner and Matthew Lyon, First Touchstone Edition, 1998 (published by Simon & Schuster).