As Leader of the Opposition, Douglas-Home persuaded Macleod and Powell to rejoin the Conservative front bench. Within weeks of the general election Butler retired from politics, accepting the post of Master of Trinity College, Cambridge together with a life peerage. Douglas-Home did not immediately allocate shadow portfolios to his colleagues, but in January 1965 he gave Maudling the foreign affairs brief and Heath became spokesman on Treasury and economic affairs. There was no immediate pressure for Douglas-Home to hand over the leadership to a member of the younger generation, but by early 1965 a new Conservative group called PEST (Pressure for Economic and Social Toryism) had discreetly begun to call for a change. Douglas-Home either did not know, or chose to ignore, the fact that Heath had made a donation to PEST. He decided that the time was coming for him to retire as leader, with Heath as his preferred successor.
Determined that the party should abandon the "customary processes of consultation", which had caused such rancour when he was appointed in 1963, Douglas-Home set up an orderly process of secret balloting by Conservative MPs for the election of his immediate and future successors as party leader. In the interests of impartiality the ballot was organised by the 1922 Committee, the backbench Conservative MPs. Douglas-Home announced his resignation as Conservative leader on 22 July 1965. Three candidates stood for the vacancy: Heath, Maudling and Powell. Heath won with 150 votes (one of them cast by Douglas-Home) to 133 for Maudling and 15 for Powell.
Douglas-Home accepted the foreign affairs portfolio in Heath's shadow cabinet. Many expected this to be a short-lived appointment, a prelude to Douglas-Home's retirement from politics. It came at a difficult time in British foreign relations: events in the self-governing colony of Rhodesia (formerly Southern Rhodesia), which had been drifting towards crisis for some years, finally erupted into open rebellion. The predominantly white minority government there opposed an immediate transfer to black majority rule before sovereign statehood, and in November 1965 it unilaterally declared independence. Douglas-Home won the approval of left-wing Labour MPs such as Wedgwood Benn for his unwavering opposition to the rebel government, and for ignoring those on the right wing of the Conservative party who sympathised with the rebels on racial grounds.
In 1966 Douglas-Home became president of the MCC, which was then the governing body of English and world cricket. The presidency had generally been a largely ceremonial position, but Douglas-Home became embroiled in two controversies, one of them with international implications. This was the so-called "D'Oliveira affair", in which the inclusion of a non-white player in the England team to tour South Africa led to the cancellation of the tour by the apartheid regime in Pretoria. In his account of the affair, the political journalist Peter Oborne criticises Douglas-Home for his vacillating attitude towards the South African Prime Minister, B J Vorster with whom, says Oborne, "he was no more robust than Chamberlain had been with Hitler thirty years earlier". Douglas-Home's advice to the MCC committee not to press the South Africans for advance assurances on D'Oliveira's acceptability, and his optimistic assurances that all would be well, became a matter of much criticism from a group of MCC members led by the Rev David Sheppard. The second controversy was not one of race but of social class. Brian Close was dropped as England captain in favour of Colin Cowdrey. Close was dropped after using delaying tactics when captaining Yorkshire in a county match, but the move was widely seen as biased towards cricketers from the old amateur tradition, which had officially ended in 1963.
Wilson's small majority after the 1964 general election had made the transaction of government business difficult, and in 1966 he called another election in which Labour gained a strong working majority of 96. Some older members of Heath's team, including Lloyd, retired from the front bench, making room for members of the next generation. Heath moved Maudling to the foreign affairs portfolio, and Douglas-Home took over Lloyd's responsibilities as spokesman on Commonwealth relations. Heath was widely seen as ineffective against Wilson, and as the 1970 general election approached there was concern within the party that he would lose, and that Powell would seek to replace him as leader. Maudling and the chief whip, William Whitelaw, believed that if Heath had to resign Douglas-Home would be the safest candidate to keep Powell out. Douglas-Home shared their view that Labour would win the 1970 election, and that Heath might then have to resign, but he declined to commit himself. To the surprise of almost everyone except Heath, the Conservatives won the election, with a majority of 31 seats.