Trevor Kennedy, friend of Australia’s high and mighty, and wealthy entrepreneur, had it all. But under investigation for his part in the Swiss bank affair involving the flamboyant and imprisoned broker Rene Rivkin, self-confidence and some friends have deserted him.

Trevor Kennedy’s relationship with disgraced stockbroker Rene Rivkin has cast his name in infamy. Rivkin’s assertion in an interview with a Swiss district attorney that Kennedy held part of a mystery parcel of shares in the controversial Offset Alpine has jeopardised everything. In the public imagination, Kennedy is inextricably linked to Rivkin’s calamitous and bizarre fall from grace, a story that threatens to run unresolved for years to come.

It was 1964 and Kennedy had sworn off journalism for a career as a potato farmer. It made financial sense, for journalism was not the career you went into to make money in those days. Though he had aspired to journalism, there was always an internal struggle: a desire to be rich. At 13, Kennedy had held a professional fisherman’s licence but had instead taken up a cadetship with his local newspaper. Later, he had become a star reporter on The Canberra Times, taking pride in how he was never intimidated by the high and mighties he interviewed.

Today Kennedy, the businessman, again finds himself in a predicament largely of his own making. The regret and humiliation are written all over his face as I meet him at his office in Sydney’s Macquarie Street. He has decided his is the view from the end of the world. His eyes are downcast as he speaks of his isolation, the frustration of having to prove his innocence to the world before he’s even charged with an offence. It’s not all bad, though: he’s lost quite a few kilos from his stocky frame. There’s not as many invitations to lunch these days perhaps. Close friends have remained loyal to Kennedy and his wife, Christina, but at social functions they feel the need to clear the air, to explain themselves, says one long-time associate. Kennedy himself is spending a lot of time at Horse Island, the private retreat on the NSW south coast he built with Christina, the daughter of a prominent Sydney medical family.

There are those who feel betrayed by the revelations that Kennedy, or entities controlled by him, operated Swiss bank accounts or engaged in other offshore transactions. One former business associate, who declined to be named or lend his support to Kennedy, suggested it would be inadvisable to delve into Kennedy’s network of mates. “This is a dangerous story … If guilt by association were a crime, then there wouldn’t be enough jails to hold us all,” he told The Bulletin.

When Rivkin was trying to avoid jail on charges of insider trading last year, Kennedy was one of few prepared to refer to him as “my friend Rene”. At Rivkin’s sentencing last year, Kennedy gave a character reference, saying that his long-time associate’s “respect for both law and moral code [was] flawless”. I hear later from a friend that Kennedy now calls him “that madman Rivkin”.

At the heart of Kennedy’s troubles lies a series of transactions with which his links are yet uncertain or largely imputed. One long-time associate says the entire affair will yield little more than “a tax problem” for Kennedy. The balances held in Switzerland total no more than $4m. “It’s birdseed,” says one supporter, giving insight into what kind of a bird Kennedy has become in the past decade as a private investor and entrepreneur. So why has Kennedy fought the regulators so hard in opposing the handover of documents and computer hard-drives that would presumably hold no fears for him?

Kennedy has told supporters that he is doing nothing more than protecting a fundamental right to privacy, that his business affairs are his alone.