PM Pedro Sánchez's party polled 29% and will need the help of either left-wing Podemos and regional parties, or the centre right, to form a government.
For the first time since military rule ended in the 1970s, a far-right party is set to enter parliament.
Vox opposes multiculturalism, feminism and unrestricted migration.
The other big story of the election was the collapse in support for the Popular Party (PP), which governed Spain until it was dumped from power in May 2018 in a no-confidence vote.
In its worst election ever, the PP won just 66 seats, down from 137 in the previous parliament.
The Socialists won 123 seats while their former coalition partner, Podemos, won 42.
The result is a personal success for the prime minister, who increased his party's share from 23% of the vote in 2016.
But it still leaves the Socialists and Podemos 11 seats short of the necessary 176 for a majority.
Mr Sánchez could make up the numbers with smaller regional parties, including Basque nationalists, but he is likely to need the help of Catalan pro-independence parties, which withdrew their support for his government last month, forcing the election.
The centre-left Catalan ERC was the big winner in Catalonia, with a projected 15 seats. Its leader, Oriol Junqueras, is in jail facing trial for his role in declaring independence in October 2017 and tweeted thanks for the million votes his party received.