Backing the initiative are social media giants, including Facebook, Twitter, YouTube and Microsoft.
The new code means that illegal hate speech could be removed from sites and other forums in less than 24 hours.
Hate speech is defined by the Commission as something that "criminalises public incitement to violence or hatred against a group of persons or a member of such a group defined by reference to race, colour, religion, descent or national or ethnic origin."
Announcing the pact at a news conference, Vĕra Jourová, European justice, consumers and gender equality Commissioner, said: "The internet is a place for free speech, not hate speech.
"The code of conduct against illegal online hate speech, agreed with IT companies, will ensure that public incitement to violence to hatred has no place online. I welcome the commitment of worldwide IT companies to review the majority of valid notifications for removal of illegal hate speech in less than 24 hours and remove or disable access to such content, if necessary."
Further comment came from Twitter's head of public policy for Europe, Karen White, who said, "Hateful conduct has no place on Twitter and we will continue to tackle this issue head on alongside our partners in industry and civil society.
"We remain committed to letting the tweets flow. However, there is a clear distinction between freedom of expression and conduct that incites violence and hate. In tandem with actioning hateful conduct that breaches Twitter's rules, we also leverage the platform's incredible capabilities to empower positive voices, to challenge prejudice and to tackle the deeper root causes of intolerance."
White added; "We look forward to further constructive dialogue between the European Commission, member states, our partners in civil society and our peers in the technology sector on this issue."
By signing the code of conduct, IT companies said they were "committed to tackling quickly and efficiently illegal hate speech online."
This will include the establishment of internal procedures and staff training to guarantee that a majority of illegal content is assessed and, where necessary, removed within 24 hours.
IT companies have agreed to strengthen their partnerships with civil society organisations in flagging content that promotes incitement to violence and hateful conduct.
The partnership will also support civil society organisations in delivering effective anti-hate campaigns to countering hateful rhetoric online.
All parties say they will assess the public commitments in this code of conduct on a regular basis, including their impact.
They also agree to further discuss how to "promote transparency and encourage counter and alternative narratives."
Regular meetings will take place and a preliminary assessment will be reported to the high level group on combating racism, xenophobia and all forms of intolerance by the end of 2016.
Reacting to the news, Monika Bickert, head of global policy management at Facebook said: "We welcome today's announcement and the chance to continue our work with the Commission and wider tech industry to fight hate speech."
Elsewhere, John Frank, vice president EU government affairs at Microsoft, added: "We value civility and free expression, and so our terms of use prohibit advocating violence and hate speech on Microsoft-hosted consumer services."
Google's public policy and government relations director, Lie Junius, said: "We're committed to giving people access to information through our services, but we have always prohibited illegal hate speech on our platforms."