Phillip was born in Aberdeen on 19 April 1817. Phillip's artistic talent was recognised at an early age. Lord Panmure paid for Phillip to become the student of Thomas Musgrave Joy in London briefly in 1836. His education at the Royal Academy of Arts was paid for by Panmure.

While at the academy, Phillip became a member of The Clique, a group of aspirant artists organised by Richard Dadd. The Clique identified as followers of  William Hogarth and David WIlkie. Phillip's own career was to follow that of fellow-Scot Wilkie very closely, beginning with carefully detailed paintings depicting the lives of Scottish crofters. He moved on to much more broadly painted scenes of Spanish life influenced by Bartolomé Esteban Murillo and Diego Velázquez.

Phillip's early works tended to depict pious Scots families. In 1851, he visited Spain after he was advised to travel to southern Europe for his health. Thereafter he concentrated on Spanish subjects. The first of these, The Letter Writer, Seville, displayed the influence of Pre-Raphaelitism, a movement he had previously opposed, along with most other members of The Clique, despite his friendship with Millais, one of its leaders. He was so influenced by his travels that he advised other artists to do the same. Some artists, such as Edwin Long, took this advice and were similarly inspired.

In the late 1850s and 1860s, Phillip's style became much broader and more painterly, in line with Millais's late work. Phillip's two most important paintings in these years were The Early Career of Murillo (1864) and La Gloria (1865, National Gallery of Scotland). The first depicted the young Murillo drawing his art from Spanish street-life; the second portrayed a Spanish wake for a dead child. Phillip was commissioned to paint the wedding in 1858 of Victoria, Princess Royal to Prince Frederick William of Prussia, later German Emperor Frederick III.

Phillip married Richard Dadd's sister. Like her brother she became insane.

Phillip's self-portrait, The Evil Eye, commissioned by his close friend Patrick Allan-Fraser, is in Hospitalfield House in Arbroath, along with portraits of other members of The Clique.

In 1867, Phillip died on 27 February 1864, aged 49, of a stroke while visiting William Powell Frith in Kensington and is buried in Kensal Green Cemetery, behind the grand monument to William Mulready on the central avenue.