It's a season of retrospection for Bruce Springsteen. New light has been thrown on his pivotal 1978 album Darkness on the Edge of Town with the release of The Promise, a double CD of out-takes and unreleased songs, alongside an expanded box set of CDs and DVDs telling the Darkness story in sound and vision. A version of Thom Zimny's documentary about the making of the album, included in the boxed release, was shown in Imagine on BBC One.

Anyone who has ever spent even a little time in a recording studio will be aware that the process of making an album lies somewhere between “watching paint dry” and “ripping out your own toenails” on the scale of interesting and enjoyable activities. It rarely makes for great television. The first image we saw in last night’s Imagine was of a youthful Bruce Springsteen holed up in New York’s Record Plant studio in 1977. He yawned; then he yawned again. Here we go, I thought.

What elevated the film to more than just muso musing about “sound pictures”, “dead rooms” and “snare sounds”, all of which reaffirmed the truism that making records is generally about as much fun as dental extraction, were the uniquely dramatic circumstances it documented. This programme was an edited version of the Thom Zimny documentary The Promise: The Making of Darkness on the Edge of Town, which accompanies the boxed version of The Promise, the recently released double album comprising 21 songs recorded in 1977 and 1978 by Springsteen during the making of his fourth album, Darkness on the Edge of Town.

Recalled through a mixture of archive footage and new interviews with all those involved, these epic sessions held a significance beyond their immediate context. They soundtracked an artist in a state of personal, professional and creative flux. Specifically, the Darkness... sessions took place beneath two hovering storm clouds – one was fame, the other was a lawsuit. It was obvious that Springsteen found the former by far the more troublesome.

Having roared to stardom in 1975 with Born to Run, he was battling what he called in the film “the separation of success”. On his guard against accusations of frivolity and hype, Springsteen resolved that his next album would be “a reaction to my own good fortune, reflecting a sense of accountability to the people I grew up with”. His first three records had been wild, boisterous, theatrical affairs filled with carnival music, urban gypsies and romanticised street characters. With Darkness... he wanted to drain all that colour away, leaving only what he described as “an austere, apocalyptic grandeur”. Or as his manager and producer Jon Landau put it: “We wanted the coffee black.” 

His new songs were a reckoning with the adult world of work, compromise and disappointment. Interviewed in 2010 for the film, Springsteen said he asked himself: how do we honour our own lives? What can and cannot be compromised without losing yourself? These questions were especially pertinent given the fact that at the time he was embroiled in legal action with his manager, Mike Appel, which boiled down to the question of who had creative control over Springsteen's career.

While the lawsuit was ongoing, he was prevented from going into the studio with any producer not approved by Appel. So at first he simply didn’t go in at all. We saw some fantastic footage shot in 1977 at his New Jersey farm, when he was effectively under the recording equivalent of house arrest. Stripped to the waist, sporting a hairstyle apparently modelled on Bob Dylan’s dog, he looked like some creative outlaw on the lam. Interestingly, he seemed to positively embrace the lawsuit. It made him an outsider again at a time of bewildering success, and steeled his resolve to follow his vision without compromise.

In June 1977 the suit was settled in Springsteen’s favour and he entered Record Plant in New York to begin recording. We saw that process evolve through old black-and-white film (pictured above) depicting long hours, days and weeks of frustration, mechanical drudgery and confusion, punctuated by some brief, electrifying moments of pure musical connection - none more so than the joyous run through of “Sherry Darling”, with Springsteen bashing out the chords on the piano and his guitarist Steve Van Zandt hammering out a rhythm with a pair of drum sticks on what looked like a rolled up carpet.

• Bruce Springsteen: In His Own Words reviewed on theartsdesk

Overleaf: watch "Sherry Darling" performed on The Promise: The Making of Darkness On the Edge of Town