Behind the parting of IBM and Blue Waters

Behind the parting of IBM and Blue Waters

CHAMPAIGN — More than three dozen changes, most suggested by IBM, would have delayed the Blue Waters project by a year, the University of Illinois argued in documents obtained by The News-Gazette.

The requested changes caused friction as early as December 2010, eight months before IBM pulled out, leaving the project to look for a new vendor for the supercomputer.

Documents released under the Freedom of Information Act show Big Blue and the Big U asserting their rights in lengthy and increasingly testy, but always polite, language.

In the documents, IBM suggested that if changes were not made, the project would become overly expensive.

They also show that talks on the water-cooled supercomputer project were breaking down at least as early as December 2010, with IBM asking for changes that the Blue Waters leaders said would hurt performance and delay the project.

The project is expected to cost more than $300 million, with two-thirds of that from the National Science Foundation and a third from the state of Illinois.

Any proprietary information that would show the specifications IBM wished to have changed has been blacked out. Nor was NCSA or IBM willing to discuss that information in interviews.

Blue Waters was to have a steady speed of 1 petaflop — a thousand trillion floating point operations per second.

The maximum of 10 petaflops was to give the University of Illinois' National Center for Supercomputing Applications and the National Science Foundation one of the fastest supercomputers in the world.

But IBM pulled out of the project in August, and NCSA and the National Science Foundation are scrambling for a new vendor to keep ahead of rapid changes in the industry.

John Melchi, a senior associate director at NCSA, said last week that there is a variety of vendors available, which he compared to a choice of car dealers.

IBM ended its relationship with the Blue Waters project as a prime supplier of the water-cooled supercomputer effective Aug. 6. IBM returned its $30 million fee to NCSA a week later. Melchi said then that "per our plan, IBM was here uninstalling the three racks (of Power7 servers)."

Melchi told The News-Gazette last week that by August, NCSA asked "what is in the best interest of the project."

"Allowing IBM to terminate ... it became clear that it was in our best interests to allow IBM to do so, because it was unable to meet the detailed performance metrics specified in the contract," Melchi said.

An IBM spokeswoman said that Big Blue had tried to accommodate Blue Waters' needs.

"IBM and the NCSA worked closely on proposals to retain IBM's participation as the supercomputer vendor, but could not come to a mutually agreed-upon plan concerning the path forward. IBM exercised its rights under the contract to terminate," IBM spokeswoman Joanna Brewer said.

She explained the corporation's refusal to reveal technological questions by saying "Each supercomputer project undertaken by IBM has its own unique characteristics and specifications. We are not discussing the specifics of the Blue Waters project."

But she did say the project was becoming too expensive.

Though she declined to answer technical questions, the FOIA documents mention clock speed as an issue.

"The project selected IBM in 2007 as the supercomputer vendor for the Blue Waters project based on projections of future technology development," Brewer said. "The innovative technology that IBM ultimately developed was more complex and required significantly increased financial and technical support by IBM beyond its original expectations. We are not going into specifics."

In a Feb. 7 email, IBM said the deadline of fall 2011 was still feasible. But questions of changes that IBM contended would make the project too expensive continued to come up in the correspondence.

A March 28 letter from Ravi Iyer, interim vice chancellor for research, said the UI was amenable to some of IBM's desired changes, but not others.

In a March 31 letter from NCSA director Thom Dunning to Brian Connors, vice president for the High Performance Computing Business Line of IBM, there was a sympathetic but firm denial of a change request from IBM.

The Blue Waters Change Control Board met and rejected at least one change request "associated with IBM's proposed change in the delivery schedule for Blue Waters," Dunning wrote. "Although the board acknowledged that many of the changes were due to factors related to development delays and was sympathetic to the changes requested by IBM, they recommended that the (request) be denied."

Dunning offered two possibilities for a mutually acceptable change, also blacked out in the FOIA documents.

On the next day, Iyer wrote:

"This is an independent board of senior faculty and researchers. There is real concern that IBM may end (up) not providing any compensation," and suggested wording changes.

Dunning wrote Connors on April 6 to further explain the rejection of the change:

"The change request delays the delivery of Blue Waters by a year, as compared to the original request. While we understand that the original schedule is not achievable any more, we are also keenly aware that the one-year delay reduces the value of the system to the research community, NSF and the University of Illinois, and increases the expenses incurred by the University and NSF.

"While one can debate the magnitude of this damage, and the proper compensation for it, it does not seem reasonable to us to accept a change to the schedule with no compensation (e.g., in forms of additional equipment) for the reduced value of the system."

He told IBM that if it were to submit a new change request that met the needs of NCSA and NSF, the institutions would grant the request and delay a discussion of compensation.

Four days later, another NCSA official, Deputy Director William Kramer, emailed IBM's Curtis Vinson that the response to the April 6 memo was "not acceptable" and did not address the changes needed.

On April 22, Dunning wrote Rod Adkins, a senior vice president of the IBM Systems and Technology Group, that, "for the past year, the delivery schedule for the Power7-IH system has suffered a day for day slippage, delaying the expected completion date for having the system ready for science use from June 2011 to June 2012.

"As of this date, IBM has not agreed to a new schedule nor has it agreed to address the issue of the change in value of the system resulting from the one-year delay. The project variances, which are reported to the federal government, are now reflecting this fact."

Dunning asked for a meeting, noting that "While Illinois has been supportive of past change proposals, and worked with IBM to reach mutually acceptable compromises, the current approach of seeking only unilateral IBM benefits and shifting major risks and costs all to the University and NSF are not reasonable or acceptable."

IBM responded April 25, in a letter formally notifying NCSA of its desire to terminate the "statement of work," IBM's official statement on how the work would be executed.

Connors wrote that the statement of work was "not feasible" and required additional expenditure by his company. He asked to begin dispute resolution.

On May 2, interim Chancellor Robert Easter wrote to Connors, saying he was disappointed at this juncture after years of working with IBM on projects.

Easter contended that terminating the statement of work was "only a last resort" after conflict resolution.

He wrote that NCSA had agreed to 36 changes since 2008, "the vast majority proposed by or for the benefit of IBM."

The chancellor then called on IBM not to renege on its promises, noting the project relied on IBM's reputation for success in choosing it as a vendor.

By June 16, Easter showed some optimism that a meeting and a teleconference had smoothed out some of the bugs.

But he wrote IBM Vice President Joseph Benaroya in more forceful terms on July 6. The statement of work "is without question a firm fixed price contract that can only be modified by the mutual agreement of the parties," Easter wrote.

He continued that rules set up for dispute resolution do not allow for unilateral action. Easter also argued that with time restraints it was not the time for "lengthy legal arguments."

Further, "we have hardly reached the type of 'last resort' from which IBM could arguably terminate the contract in good faith," Easter wrote.

Were IBM unilaterally to withdraw from the contract, it would still face substantial penalties, he added. The UI and NSF had not received substantial benefits from "scant materials IBM has provided to date."

On Aug. 6, IBM and NCSA announced that IBM was withdrawing from the project.