Born to wealthy parents, Broughton lost his father early to the 1918 influenza epidemic and spent the rest of his life getting over his high-strung, overbearing mother.
Before he was three, "Sunny Jim" experienced a transformational visit from his muse, Hermy, which he describes in his autobiography, Coming Unbuttoned (1993):
I remember waking in the dark and hearing my parents arguing in the next room. But a more persistent sound, a kind of whirring whistle, spun a light across the ceiling. I stood up in my crib and looked into the backyard. Over a neighbor’s palm tree a pulsing headlamp came whistling directly toward me. When it had whirled right up to my window, out of its radiance stepped a naked boy. He was at least three years older than I but he looked all ages at once. He had no wings, but I knew he was angel-sent: his laughing beauty illuminated the night and his melodious voice enraptured my ears….
He insisted I would always be a poet even if I tried not to be….Despite what I might hear to the contrary the world was not a miserable prison, it was a playground for a nonstop tournament between stupidity and imagination. If I followed the game sharply enough, I could be a useful spokesman for Big Joy.
In the book, Broughton remarks on his love affairs with both men and women. Among his male lovers was gay activist Harry Hay.
He briefly lived with the film critic Pauline Kael and they had a daughter, Gina, who was born in 1948.Broughton is the subject of the 2012 award-winning documentary film, Big Joy: the adventures of James Broughton from Stephen Silha, Eric Slade, Dawn Logson and cinematographer Ian Hinkle.