Bruce Weber first became interested in Chet Baker when he spotted a photograph of the musician in a Pittsburgh record store on the cover of the 1955 vinyl LP Chet Baker Sings and Plays with Bud Shank, Russ Freeman and Strings when he was 16-years-old. Weber first met Baker in the winter of 1986 at a club in New York City and convinced him to do a photo shoot and what was originally only going to be a three-minute film. Weber had wanted to make a short film from an Oscar Levant song called "Blame It on My Youth". They had such a good time together that Baker started opening up to Weber. Afterwards, Weber convinced Baker to make a longer film and the musician agreed. Filming began in January 1987. Interviewing Baker was a challenge as Weber remembers, "Sometimes we'd have to stop for some reason or another and then, because Chet was a junkie and couldn't do things twice, we'd have to start all over again. But we grew to really like him".
In May 1987, when Weber's documentary Broken Noses premiered at the Cannes Film Festival, he brought Baker along to shoot footage for Let's Get Lost. Weber spent a million dollars of his own money on the documentary and filmed it when he had the time and the money, describing it as a "a very ad hoc film". The film's title comes from a song performed by Baker and recorded on the album Chet Baker Sings and Plays, which was the first Baker album director Bruce Weber bought when he was 16-years-old at a Pittsburgh record store.